{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quizbowl-the-case-for-incremental-question","title":"Quizbowl: The Case for Incremental Question Answering","arxiv_id":"1904.04792","date":"2019-04-09","proceeding":null,"authors":["Pedro Rodriguez","Shi Feng","Mohit Iyyer","He He","Jordan Boyd-Graber"],"abstract":"Scholastic trivia competitions test knowledge and intelligence through mastery of question answering. Modern question answering benchmarks are one variant of the Turing test. Specifically, answering a set of questions as well as a human is a minimum bar towards demonstrating human-like intelligence. This paper makes the case that the format of one competition -- where participants can answer in the middle of hearing a question (incremental) -- better differentiates the skill between (human or machine) players. Additionally, merging a sequential decision-making sub-task with question answering (QA) provides a good setting for research in model calibration and opponent modeling. Thus, embedded in this task are three machine learning challenges: (1) factoid QA over thousands of Wikipedia-like answers, (2) calibration of the QA model's confidence scores, and (3) sequential decision-making that incorporates knowledge of the QA model, its calibration, and what the opponent may do. We make two contributions: (1) collecting and curating a large factoid QA dataset and an accompanying gameplay dataset, and (2) developing a model that addresses these three machine learning challenges. In addition to offline evaluation, we pitted our model against some of the most accomplished trivia players in the world in a series of exhibition matches spanning several years. Throughout this paper, we show that collaborations with the vibrant trivia community have contributed to the quality of our dataset, spawned new research directions, and doubled as an exciting way to engage the public with research in machine learning and natural language processing.","url_abs":"https://arxiv.org/abs/1904.04792v2","url_pdf":"https://arxiv.org/pdf/1904.04792v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quizbowl-the-case-for-incremental-question","repo_url":"https://github.com/h-j-han/automatic_explicitation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"}],"methods":[],"datasets_introduced":[{"slug":"quizbowl","name":"Quizbowl","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.04792","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1904.04792"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/h-j-han/automatic_explicitation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":14},"by_repo_kind":{"listed":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ee20d0700e2f61f7","entry":"add_highlight_tokidx","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"show.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/show.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee20d0700e2f61f7"}},{"code_sha256_prefix":"a3254eef985754fe","entry":"create_tokenizer","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/tokenizer.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3254eef985754fe"}},{"code_sha256_prefix":"3c2ffcbb4d795cb4","entry":"create_tokenizer4pairs","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/tokenizer.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c2ffcbb4d795cb4"}},{"code_sha256_prefix":"d501e482699c498b","entry":"find_sent_idx","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/xqb/display_explicitation.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/xqb/display_explicitation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d501e482699c498b"}},{"code_sha256_prefix":"9820556ecac3062a","entry":"get_token_char_indices","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/utils.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9820556ecac3062a"}},{"code_sha256_prefix":"c1968e7cd9190083","entry":"get_unaligned_idx","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/utils.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1968e7cd9190083"}},{"code_sha256_prefix":"99b73eb7e1a4905d","entry":"group_ner","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/ner.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/ner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99b73eb7e1a4905d"}},{"code_sha256_prefix":"fe80f19e05d4e058","entry":"insert_token_make_batchsents","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/entitylink.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/entitylink.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe80f19e05d4e058"}},{"code_sha256_prefix":"cca6327af9af1e77","entry":"plot3","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/plots.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/plots.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cca6327af9af1e77"}},{"code_sha256_prefix":"1b161f90a980b964","entry":"plot6","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/plots.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/plots.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b161f90a980b964"}},{"code_sha256_prefix":"5983071885cc410d","entry":"plot8","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/plots.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/plots.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5983071885cc410d"}},{"code_sha256_prefix":"8c9acd0ab7ead2bd","entry":"print_span","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"show.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/show.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c9acd0ab7ead2bd"}},{"code_sha256_prefix":"8d1d41b003e7a86e","entry":"remove_duplicate_tuples","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"autoexpl/utils.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/autoexpl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d1d41b003e7a86e"}},{"code_sha256_prefix":"f84f0926a1f7a503","entry":"tup_range_to_idx","repo":"h-j-han/automatic_explicitation","repo_kind":"listed","path":"show.py","file_url":"https://github.com/h-j-han/automatic_explicitation/blob/HEAD/show.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f84f0926a1f7a503"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}