{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/think-you-have-solved-question-answering-try","title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","arxiv_id":"1803.05457","date":"2018-03-14","proceeding":null,"authors":["Peter Clark","Isaac Cowhey","Oren Etzioni","Tushar Khot","Ashish Sabharwal","Carissa Schoenick","Oyvind Tafjord"],"abstract":"We present a new question set, text corpus, and baselines assembled to\nencourage AI research in advanced question answering. Together, these\nconstitute the AI2 Reasoning Challenge (ARC), which requires far more powerful\nknowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC\nquestion set is partitioned into a Challenge Set and an Easy Set, where the\nChallenge Set contains only questions answered incorrectly by both a\nretrieval-based algorithm and a word co-occurence algorithm. The dataset\ncontains only natural, grade-school science questions (authored for human\ntests), and is the largest public-domain set of this kind (7,787 questions). We\ntest several baselines on the Challenge Set, including leading neural models\nfrom the SQuAD and SNLI tasks, and find that none are able to significantly\noutperform a random baseline, reflecting the difficult nature of this task. We\nare also releasing the ARC Corpus, a corpus of 14M science sentences relevant\nto the task, and implementations of the three neural baseline models tested.\nCan your model perform better? We pose ARC as a challenge to the community.","url_abs":"http://arxiv.org/abs/1803.05457v1","url_pdf":"http://arxiv.org/pdf/1803.05457v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"think-you-have-solved-question-answering-try","repo_url":"https://github.com/meetyou-ai-lab/can-mc-evaluate-llms","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"AI2 Reasoning Challenge"},{"task_slug":"arc","task_name":"ARC"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"arc","name":"ARC (AI2 Reasoning Challenge)","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1803.05457","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}