{"url":"/dataset/arc-da","name":"ARC-DA","full_name":"ARC Direct Answer Questions","description_markdown":"**ARC Direct Answer Questions** (**ARC-DA**) dataset consists of 2,985 grade-school level, direct-answer (\"open response\", \"free form\") science questions derived from the ARC multiple-choice question set released as part of the AI2 Reasoning Challenge in 2018.\r\n\r\n### How the dataset was built\r\nThese questions were derived from the ARC multiple-choice question set released as part of the AI2 Reasoning Challenge in 2018. The ARC Easy and ARC Challenge set questions in the original dataset were combined and then filtered/modified by the following process:\r\n\r\n- Turking: Each of the multiple-choice questions was presented as a direct answer question to five crowdsourced workers to gather additional answers.\r\n\r\n- Heuristic filtering: The questions were filtered based on the following heuristic filters:\r\n    - Questions having a threshold number of turker answers, as a proxy for concreteness of the question.\r\n    - Questions having at least two turker-provided answers with word overlap, as a measure of confidence in the correctness of the answers, and also straightforwardness of the question.\r\n    - Other heuristics to identify questions that only make sense as multiple-choice questions, such as, questions starting with the phrase “Which of the following”.\r\n\r\n- Further manual vetting: We had volunteers in house do another pass of vetting where they:\r\n     - Marked highly open-ended questions with too many answer choices, such as “Name an insect”, or otherwise invalid questions, for removal. These are filtered out.\r\n     - Removed some of the bad answers gathered from turking.\r\n     - Reworded questions to make them more suited to direct answer question format, for e.g., a question such as “What element is contained in table salt?” which would make sense as a multiple-choice question, needs be reworded to something like “Name an element present in table salt”.\r\n     - Added any additional answers to the questions they could think of that were not present in the turker provided answers.\r\n\r\nImage source: [ARC-DA dataset](https://allenai.org/data/arc-da)","description_withheld":null,"homepage":"https://allenai.org/data/arc-da","introduced_date":"2021-02-05","introduced_date_note":null,"introduced_by":{"paper":"/paper/think-you-have-solved-direct-answer-question","title":"Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge","first_author":"Sumithra Bhakthavatsalam","url":null},"license":null,"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["ARC-DA"],"data_loaders":[],"num_papers_in_archive":4,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}