{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/carpe-diem-on-the-evaluation-of-world","title":"Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models","arxiv_id":"2311.08106","date":"2023-11-14","proceeding":null,"authors":["Yujin Kim","Jaehong Yoon","Seonghyeon Ye","Sangmin Bae","Namgyu Ho","Sung Ju Hwang","Se-Young Yun"],"abstract":"The dynamic nature of knowledge in an ever-changing world presents challenges for language models trained on static data; the model in the real world often requires not only acquiring new knowledge but also overwriting outdated information into updated ones. To study the ability of language models for these time-dependent dynamics in human language, we introduce a novel task, EvolvingQA, a temporally evolving question-answering benchmark designed for training and evaluating LMs on an evolving Wikipedia database. The construction of EvolvingQA is automated with our pipeline using large language models. We uncover that existing continual learning baselines suffer from updating and removing outdated knowledge. Our analysis suggests that models fail to rectify knowledge due to small weight gradients. In addition, we elucidate that language models particularly struggle to reflect the change of numerical or temporal information. Our work aims to model the dynamic nature of real-world information, suggesting faithful evaluations of the evolution-adaptability of language models.","url_abs":"https://arxiv.org/abs/2311.08106v2","url_pdf":"https://arxiv.org/pdf/2311.08106v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"carpe-diem-on-the-evaluation-of-world","repo_url":"https://github.com/kimyuji/evolvingqa_benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.08106","atlas_url":"https://app.syntology.ai/?focus=2311.08106","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.08106"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kimyuji/evolvingqa_benchmark","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"b8301a2a061e30a2","entry":"anneal_function","repo":"kimyuji/evolvingqa_benchmark","repo_kind":"official","path":"models/RecAdam.py","file_url":"https://github.com/kimyuji/evolvingqa_benchmark/blob/HEAD/models/RecAdam.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8301a2a061e30a2"}},{"code_sha256_prefix":"136abfa3e8ce028d","entry":"load_tf_weights_in_t5","repo":"kimyuji/evolvingqa_benchmark","repo_kind":"official","path":"models/T5_Model_Kadapter.py","file_url":"https://github.com/kimyuji/evolvingqa_benchmark/blob/HEAD/models/T5_Model_Kadapter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"136abfa3e8ce028d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}