{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adversarial-examples-for-evaluating-reading","title":"Adversarial Examples for Evaluating Reading Comprehension Systems","arxiv_id":"1707.07328","date":"2017-07-23","proceeding":"EMNLP 2017 9","authors":["Robin Jia","Percy Liang"],"abstract":"Standard accuracy metrics indicate that reading comprehension systems are\nmaking rapid progress, but the extent to which these systems truly understand\nlanguage remains unclear. To reward systems with real language understanding\nabilities, we propose an adversarial evaluation scheme for the Stanford\nQuestion Answering Dataset (SQuAD). Our method tests whether systems can answer\nquestions about paragraphs that contain adversarially inserted sentences, which\nare automatically generated to distract computer systems without changing the\ncorrect answer or misleading humans. In this adversarial setting, the accuracy\nof sixteen published models drops from an average of $75\\%$ F1 score to $36\\%$;\nwhen the adversary is allowed to add ungrammatical sequences of words, average\naccuracy on four models decreases further to $7\\%$. We hope our insights will\nmotivate the development of new models that understand language more precisely.","url_abs":"http://arxiv.org/abs/1707.07328v1","url_pdf":"http://arxiv.org/pdf/1707.07328v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adversarial-examples-for-evaluating-reading","repo_url":"https://worksheets.codalab.org/worksheets/0xc86d3ebe69a3427d91f9aaa63f7d1e7d","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"adversarial-examples-for-evaluating-reading","repo_url":"https://github.com/SatyamSoni23/Smart-Question-Answering-System-on-Document","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"adversarial-examples-for-evaluating-reading","repo_url":"https://github.com/makcedward/nlpaug","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1707.07328","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}