{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quasar-datasets-for-question-answering-by","title":"Quasar: Datasets for Question Answering by Search and Reading","arxiv_id":"1707.03904","date":"2017-07-12","proceeding":null,"authors":["Bhuwan Dhingra","Kathryn Mazaitis","William W. Cohen"],"abstract":"We present two new large-scale datasets aimed at evaluating systems designed\nto comprehend a natural language query and extract its answer from a large\ncorpus of text. The Quasar-S dataset consists of 37000 cloze-style\n(fill-in-the-gap) queries constructed from definitions of software entity tags\non the popular website Stack Overflow. The posts and comments on the website\nserve as the background corpus for answering the cloze questions. The Quasar-T\ndataset consists of 43000 open-domain trivia questions and their answers\nobtained from various internet sources. ClueWeb09 serves as the background\ncorpus for extracting these answers. We pose these datasets as a challenge for\ntwo related subtasks of factoid Question Answering: (1) searching for relevant\npieces of text that include the correct answer to a query, and (2) reading the\nretrieved text to answer the query. We also describe a retrieval system for\nextracting relevant sentences and documents from the corpus given a query, and\ninclude these in the release for researchers wishing to only focus on (2). We\nevaluate several baselines on both datasets, ranging from simple heuristics to\npowerful neural models, and show that these lag behind human performance by\n16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at\nhttps://github.com/bdhingra/quasar .","url_abs":"http://arxiv.org/abs/1707.03904v2","url_pdf":"http://arxiv.org/pdf/1707.03904v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quasar-datasets-for-question-answering-by","repo_url":"https://github.com/bdhingra/quasar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"quasar-1","name":"QUASAR","full_name":"QUestion Answering by Search And Reading"},{"slug":"quasar-s","name":"QUASAR-S","full_name":"QUestion Answering by Search And Reading – Stack Overflow"},{"slug":"quasar-t","name":"QUASAR-T","full_name":"QUestion Answering by Search And Reading – Trivia"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1707.03904","atlas_url":"https://app.syntology.ai/?focus=1707.03904","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1707.03904"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bdhingra/quasar","reach":null}],"summary":{"ran_violates":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8c2b5b40e8ea29f0","entry":"exact_match","repo":"bdhingra/quasar","repo_kind":"official","path":"metric.py","file_url":"https://github.com/bdhingra/quasar/blob/HEAD/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8c2b5b40e8ea29f0"}},{"code_sha256_prefix":"263f5ef5dd96e7d8","entry":"f1_match","repo":"bdhingra/quasar","repo_kind":"official","path":"metric.py","file_url":"https://github.com/bdhingra/quasar/blob/HEAD/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"263f5ef5dd96e7d8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}