{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-dataset-for-answering-time-sensitive","title":"A Dataset for Answering Time-Sensitive Questions","arxiv_id":"2108.06314","date":"2021-08-13","proceeding":null,"authors":["Wenhu Chen","Xinyi Wang","William Yang Wang"],"abstract":"Time is an important dimension in our physical world. Lots of facts can evolve with respect to time. For example, the U.S. President might change every four years. Therefore, it is important to consider the time dimension and empower the existing QA models to reason over time. However, the existing QA datasets contain rather few time-sensitive questions, hence not suitable for diagnosing or benchmarking the model's temporal reasoning capability. In order to promote research in this direction, we propose to construct a time-sensitive QA dataset. The dataset is constructed by 1) mining time-evolving facts from WikiData and aligning them to their corresponding Wikipedia page, 2) employing crowd workers to verify and calibrate these noisy facts, 3) generating question-answer pairs based on the annotated time-sensitive facts. Our dataset poses challenges in the aspect of both temporal understanding and temporal reasoning. We evaluate different SoTA long-document QA systems like BigBird and FiD on our dataset. The best-performing model FiD can only achieve 46\\% accuracy, still far behind the human performance of 87\\%. We demonstrate that these models are still lacking the ability to perform consistent temporal reasoning. Therefore, we believe that our dataset could serve as a benchmark to develop NLP models more sensitive to temporal shifts. The dataset and code are released in~\\url{https://github.com/wenhuchen/Time-Sensitive-QA}.","url_abs":"https://arxiv.org/abs/2108.06314v5","url_pdf":"https://arxiv.org/pdf/2108.06314v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-dataset-for-answering-time-sensitive","repo_url":"https://github.com/wenhuchen/time-sensitive-qa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"bigbird","method_name":"BigBird"}],"datasets_introduced":[{"slug":"timeqa","name":"TimeQA","full_name":"Time-Sensitive QA"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2108.06314","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.06314"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wenhuchen/time-sensitive-qa","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":2,"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"084d22134d9bd7c4","entry":"compute_exact","repo":"wenhuchen/time-sensitive-qa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/wenhuchen/time-sensitive-qa/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"084d22134d9bd7c4"}},{"code_sha256_prefix":"6bef6bdd6eb44070","entry":"cross_attention_forward","repo":"wenhuchen/time-sensitive-qa","repo_kind":"official","path":"FiD/model.py","file_url":"https://github.com/wenhuchen/time-sensitive-qa/blob/HEAD/FiD/model.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6bef6bdd6eb44070"}},{"code_sha256_prefix":"575722a6543d189e","entry":"get_tokens","repo":"wenhuchen/time-sensitive-qa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/wenhuchen/time-sensitive-qa/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"575722a6543d189e"}},{"code_sha256_prefix":"6a96435eba311b08","entry":"normalize_answer","repo":"wenhuchen/time-sensitive-qa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/wenhuchen/time-sensitive-qa/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6a96435eba311b08"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}