{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revealing-the-importance-of-semantic","title":"Revealing the Importance of Semantic Retrieval for Machine Reading at Scale","arxiv_id":"1909.08041","date":"2019-09-17","proceeding":"IJCNLP 2019 11","authors":["Yixin Nie","Songhe Wang","Mohit Bansal"],"abstract":"Machine Reading at Scale (MRS) is a challenging task in which a system is given an input query and is asked to produce a precise output by \"reading\" information from a large knowledge base. The task has gained popularity with its natural combination of information retrieval (IR) and machine comprehension (MC). Advancements in representation learning have led to separated progress in both IR and MC; however, very few studies have examined the relationship and combined design of retrieval and comprehension at different levels of granularity, for development of MRS systems. In this work, we give general guidelines on system design for MRS by proposing a simple yet effective pipeline system with special consideration on hierarchical semantic retrieval at both paragraph and sentence level, and their potential effects on the downstream task. The system is evaluated on both fact verification and open-domain multihop QA, achieving state-of-the-art results on the leaderboard test sets of both FEVER and HOTPOTQA. To further demonstrate the importance of semantic retrieval, we present ablation and analysis studies to quantify the contribution of neural retrieval modules at both paragraph-level and sentence-level, and illustrate that intermediate semantic retrieval modules are vital for not only effectively filtering upstream information and thus saving downstream computation, but also for shaping upstream data distribution and providing better data for downstream modeling. Code/data made publicly available at: https://github.com/easonnie/semanticRetrievalMRS","url_abs":"https://arxiv.org/abs/1909.08041v1","url_pdf":"https://arxiv.org/pdf/1909.08041v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revealing-the-importance-of-semantic","repo_url":"https://github.com/easonnie/semanticRetrievalMRS","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"revealing-the-importance-of-semantic","repo_url":"https://github.com/dujiaxin/semanticRetrievalMRS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"fact-verification","task_name":"Fact Verification"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-retrieval","task_name":"Semantic Retrieval"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-hotpotqa","task":"Question Answering","dataset":"HotpotQA","model":"SemanticRetrievalMRS","rank_in_archive_order":44,"of":72,"metrics":{"ANS-EM":"0.453","ANS-F1":"0.573","JOINT-EM":"0.251","JOINT-F1":"0.476","SUP-EM":"0.387","SUP-F1":"0.708"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1909.08041","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.08041"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/easonnie/semanticRetrievalMRS","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dujiaxin/semanticRetrievalMRS","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":4,"unverified":11},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1},"listed":{"samples":13,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/bert_model_variances/bert_maxout_clf.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/bert_model_variances/bert_maxout_clf.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"5e7db7ffb7a92c64","entry":"doc_macro_precision","repo":"easonnie/semanticRetrievalMRS","repo_kind":"official","path":"src/evaluation/fever_scorer.py","file_url":"https://github.com/easonnie/semanticRetrievalMRS/blob/HEAD/src/evaluation/fever_scorer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e7db7ffb7a92c64"}},{"code_sha256_prefix":"2781a9591cbfdb33","entry":"fever_doc_only","repo":"easonnie/semanticRetrievalMRS","repo_kind":"official","path":"src/evaluation/fever_scorer.py","file_url":"https://github.com/easonnie/semanticRetrievalMRS/blob/HEAD/src/evaluation/fever_scorer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2781a9591cbfdb33"}},{"code_sha256_prefix":"bacc307a6797cb32","entry":"fever_sent_only","repo":"easonnie/semanticRetrievalMRS","repo_kind":"official","path":"src/evaluation/fever_scorer.py","file_url":"https://github.com/easonnie/semanticRetrievalMRS/blob/HEAD/src/evaluation/fever_scorer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bacc307a6797cb32"}},{"code_sha256_prefix":"40e9fee2e0b7e278","entry":"gelu","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/bert_model_variances/bert_maxout_clf.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/bert_model_variances/bert_maxout_clf.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40e9fee2e0b7e278"}},{"code_sha256_prefix":"ba1043dff5ce07b8","entry":"load_from_file","repo":"easonnie/semanticRetrievalMRS","repo_kind":"official","path":"src/build_rindex/build_rvindex.py","file_url":"https://github.com/easonnie/semanticRetrievalMRS/blob/HEAD/src/build_rindex/build_rvindex.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba1043dff5ce07b8"}},{"code_sha256_prefix":"292d77ea319f82b8","entry":"compute_K","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/bm25.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/bm25.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"292d77ea319f82b8"}},{"code_sha256_prefix":"d8f74d3b75b19f79","entry":"filter_ngram","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/build_wiki_rindex.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/build_wiki_rindex.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8f74d3b75b19f79"}},{"code_sha256_prefix":"019955faa9c53626","entry":"filter_word","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/build_wiki_rindex.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/build_wiki_rindex.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"019955faa9c53626"}},{"code_sha256_prefix":"7ba88a577b6a2d52","entry":"get_ngrams","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/document_analysis.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/document_analysis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ba88a577b6a2d52"}},{"code_sha256_prefix":"41f0dc9c6108d5df","entry":"get_query_result","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/bm25.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/bm25.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41f0dc9c6108d5df"}},{"code_sha256_prefix":"6d9943f9b30c55eb","entry":"get_raw_text","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/iterative_building.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/iterative_building.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d9943f9b30c55eb"}},{"code_sha256_prefix":"1df66838eb1b9b17","entry":"get_sentence_tokens","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/rindex.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/rindex.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1df66838eb1b9b17"}},{"code_sha256_prefix":"b1c91c9e3a3b88e4","entry":"normalize","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/build_wiki_rindex.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/build_wiki_rindex.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1c91c9e3a3b88e4"}},{"code_sha256_prefix":"0a343f931a447c8e","entry":"query_raw_text","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/raw_text_db.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/raw_text_db.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a343f931a447c8e"}},{"code_sha256_prefix":"e157daea0cea1f36","entry":"remove_all_anchor","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/iterative_building.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/iterative_building.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e157daea0cea1f36"}},{"code_sha256_prefix":"d008fe75170fedde","entry":"score_BM25","repo":"dujiaxin/semanticRetrievalMRS","repo_kind":"listed","path":"src/build_rindex/bm25.py","file_url":"https://github.com/dujiaxin/semanticRetrievalMRS/blob/HEAD/src/build_rindex/bm25.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d008fe75170fedde"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}