{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-a-suit-of-armor-conduct-electricity-a-new","title":"Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering","arxiv_id":"1809.02789","date":"2018-09-08","proceeding":"EMNLP 2018 10","authors":["Todor Mihaylov","Peter Clark","Tushar Khot","Ashish Sabharwal"],"abstract":"We present a new kind of question answering dataset, OpenBookQA, modeled\nafter open book exams for assessing human understanding of a subject. The open\nbook that comes with our questions is a set of 1329 elementary level science\nfacts. Roughly 6000 questions probe an understanding of these facts and their\napplication to novel situations. This requires combining an open book fact\n(e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of\narmor is made of metal) obtained from other sources. While existing QA datasets\nover documents or knowledge bases, being generally self-contained, focus on\nlinguistic understanding, OpenBookQA probes a deeper understanding of both the\ntopic---in the context of common knowledge---and the language it is expressed\nin. Human performance on OpenBookQA is close to 92%, but many state-of-the-art\npre-trained QA methods perform surprisingly poorly, worse than several simple\nneural baselines we develop. Our oracle experiments designed to circumvent the\nknowledge retrieval bottleneck demonstrate the value of both the open book and\nadditional facts. We leave it as a challenge to solve the retrieval problem in\nthis multi-hop setting and to close the large gap to human performance.","url_abs":"http://arxiv.org/abs/1809.02789v1","url_pdf":"http://arxiv.org/pdf/1809.02789v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-a-suit-of-armor-conduct-electricity-a-new","repo_url":"https://github.com/allenai/arc-solvers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"openbookqa","name":"OpenBookQA","full_name":"OBQA"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"BiLSTM max-out question-match (science fact + common knowledge fact)","rank_in_archive_order":26,"of":45,"metrics":{"Accuracy":"76.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"BiLSTM max-out question-match (WordNet + science fact)","rank_in_archive_order":33,"of":45,"metrics":{"Accuracy":"56.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"BiLSTM max-out question-match (with a science fact)","rank_in_archive_order":35,"of":45,"metrics":{"Accuracy":"55.8"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"Random chance baseline","rank_in_archive_order":45,"of":45,"metrics":{"Accuracy":"25"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1809.02789","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.02789"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/arc-solvers","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f92024b299799ce7","entry":"create_hypothesis","repo":"allenai/arc-solvers","repo_kind":"official","path":"arc_solvers/processing/convert_to_entailment.py","file_url":"https://github.com/allenai/arc-solvers/blob/HEAD/arc_solvers/processing/convert_to_entailment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f92024b299799ce7"}},{"code_sha256_prefix":"274f33301270f386","entry":"get_fitb_from_question","repo":"allenai/arc-solvers","repo_kind":"official","path":"arc_solvers/processing/convert_to_entailment.py","file_url":"https://github.com/allenai/arc-solvers/blob/HEAD/arc_solvers/processing/convert_to_entailment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"274f33301270f386"}},{"code_sha256_prefix":"660aff04fb937072","entry":"score_choice_predictions","repo":"allenai/arc-solvers","repo_kind":"official","path":"arc_solvers/processing/evaluate_predictions.py","file_url":"https://github.com/allenai/arc-solvers/blob/HEAD/arc_solvers/processing/evaluate_predictions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"660aff04fb937072"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}