{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-do-models-learn-from-question-answering","title":"What do Models Learn from Question Answering Datasets?","arxiv_id":"2004.03490","date":"2020-04-07","proceeding":"EMNLP 2020 11","authors":["Priyanka Sen","Amir Saffari"],"abstract":"While models have reached superhuman performance on popular question answering (QA) datasets such as SQuAD, they have yet to outperform humans on the task of question answering itself. In this paper, we investigate if models are learning reading comprehension from QA datasets by evaluating BERT-based models across five datasets. We evaluate models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. We find that no single dataset is robust to all of our experiments and identify shortcomings in both datasets and evaluation methods. Following our analysis, we make recommendations for building future QA datasets that better evaluate the task of question answering through reading comprehension. We also release code to convert QA datasets to a shared format for easier experimentation at https://github.com/amazon-research/qa-dataset-converter.","url_abs":"https://arxiv.org/abs/2004.03490v2","url_pdf":"https://arxiv.org/pdf/2004.03490v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-do-models-learn-from-question-answering","repo_url":"https://github.com/amazon-research/qa-dataset-converter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"what-do-models-learn-from-question-answering","repo_url":"https://github.com/amazon-science/qa-dataset-converter","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.03490","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.03490"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/qa-dataset-converter","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-research/qa-dataset-converter","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":2,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1},"listed":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20640d3656e293c0","entry":"answer_start","repo":"amazon-science/qa-dataset-converter","repo_kind":"listed","path":"newsqa/newsqa_to_squad.py","file_url":"https://github.com/amazon-science/qa-dataset-converter/blob/HEAD/newsqa/newsqa_to_squad.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20640d3656e293c0"}},{"code_sha256_prefix":"d9a2ef2b7ba09e79","entry":"answer_text","repo":"amazon-science/qa-dataset-converter","repo_kind":"listed","path":"newsqa/newsqa_to_squad.py","file_url":"https://github.com/amazon-science/qa-dataset-converter/blob/HEAD/newsqa/newsqa_to_squad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9a2ef2b7ba09e79"}},{"code_sha256_prefix":"8c01993af488673e","entry":"has_long_answer","repo":"amazon-science/qa-dataset-converter","repo_kind":"listed","path":"nq/nq_to_squad.py","file_url":"https://github.com/amazon-science/qa-dataset-converter/blob/HEAD/nq/nq_to_squad.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c01993af488673e"}},{"code_sha256_prefix":"a4f3efffaf7454f7","entry":"has_short_answer","repo":"amazon-science/qa-dataset-converter","repo_kind":"listed","path":"nq/nq_to_squad.py","file_url":"https://github.com/amazon-science/qa-dataset-converter/blob/HEAD/nq/nq_to_squad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4f3efffaf7454f7"}},{"code_sha256_prefix":"d3c04917ac91d32d","entry":"answer_index_in_document","repo":"amazon-research/qa-dataset-converter","repo_kind":"official","path":"triviaqa/triviaqa_to_squad.py","file_url":"https://github.com/amazon-research/qa-dataset-converter/blob/HEAD/triviaqa/triviaqa_to_squad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3c04917ac91d32d"}},{"code_sha256_prefix":"486151e6c8d08794","entry":"select_relevant_portion","repo":"amazon-research/qa-dataset-converter","repo_kind":"official","path":"triviaqa/triviaqa_to_squad.py","file_url":"https://github.com/amazon-research/qa-dataset-converter/blob/HEAD/triviaqa/triviaqa_to_squad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"486151e6c8d08794"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}