{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-dataset-of-information-seeking-questions","title":"A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers","arxiv_id":"2105.03011","date":"2021-05-07","proceeding":"NAACL 2021 4","authors":["Pradeep Dasigi","Kyle Lo","Iz Beltagy","Arman Cohan","Noah A. Smith","Matt Gardner"],"abstract":"Readers of academic research papers often read with the goal of answering specific questions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information-seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present QASPER, a dataset of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers, motivating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate.","url_abs":"https://arxiv.org/abs/2105.03011v1","url_pdf":"https://arxiv.org/pdf/2105.03011v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-dataset-of-information-seeking-questions","repo_url":"https://github.com/allenai/qasper-led-baseline","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"evidence-selection","task_name":"Evidence Selection"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[{"slug":"qasper","name":"QASPER","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-qasper","task":"Question Answering","dataset":"QASPER","model":"Longformer Encoder Decoder (base)","rank_in_archive_order":1,"of":1,"metrics":{"Token F1":"33.63"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2105.03011","atlas_url":"https://app.syntology.ai/?focus=2105.03011","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.03011"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/qasper-led-baseline","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1e3b76168a603017","entry":"get_references","repo":"allenai/qasper-led-baseline","repo_kind":"official","path":"scripts/evaluate_oracle_reranker.py","file_url":"https://github.com/allenai/qasper-led-baseline/blob/HEAD/scripts/evaluate_oracle_reranker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e3b76168a603017"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}