{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-llms-at-evaluating-temporal","title":"Is Your LLM Outdated? Evaluating LLMs at Temporal Generalization","arxiv_id":"2405.08460","date":"2024-05-14","proceeding":null,"authors":["Chenghao Zhu","Nuo Chen","Yufei Gao","Yunyi Zhang","Prayag Tiwari","Benyou Wang"],"abstract":"The rapid advancement of Large Language Models (LLMs) highlights the urgent need for evolving evaluation methodologies that keep pace with improvements in language comprehension and information processing. However, traditional benchmarks, which are often static, fail to capture the continually changing information landscape, leading to a disparity between the perceived and actual effectiveness of LLMs in ever-changing real-world scenarios. Our study examines temporal generalization, which includes the ability to understand, predict, and generate text relevant to past, present, and future contexts, revealing significant temporal biases in LLMs. We propose an evaluation framework, for dynamically generating benchmarks from recent real-world predictions. Experiments demonstrate that LLMs struggle with temporal generalization, showing performance decline over time. These findings highlight the necessity for improved training and updating processes to enhance adaptability and reduce biases. Our code, dataset and benchmark are available at https://github.com/FreedomIntelligence/FreshBench.","url_abs":"https://arxiv.org/abs/2405.08460v2","url_pdf":"https://arxiv.org/pdf/2405.08460v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-llms-at-evaluating-temporal","repo_url":"https://github.com/freedomintelligence/freshbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.08460","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.08460"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/freedomintelligence/freshbench","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"362cc4cc9069be45","entry":"format_data","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"script/format4score.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/script/format4score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"362cc4cc9069be45"}},{"code_sha256_prefix":"94857d11db1b7a9a","entry":"gather_result","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"script/score.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/script/score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94857d11db1b7a9a"}},{"code_sha256_prefix":"7e9119f9d1a780e9","entry":"get_bias","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"standard_name.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/standard_name.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e9119f9d1a780e9"}},{"code_sha256_prefix":"c06002f42e6be7ad","entry":"get_last_10_question_list","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"update_from_crawl.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/update_from_crawl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c06002f42e6be7ad"}},{"code_sha256_prefix":"b21ef2f2569d9b67","entry":"get_score","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"script/score.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/script/score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b21ef2f2569d9b67"}},{"code_sha256_prefix":"e6da4e59e0382a4e","entry":"get_target","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"GoodJudgeOpen_crawler/make_json_for_harness.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/GoodJudgeOpen_crawler/make_json_for_harness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6da4e59e0382a4e"}},{"code_sha256_prefix":"8e697eb70078ba08","entry":"parse_possible_name","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"standard_name.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/standard_name.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e697eb70078ba08"}},{"code_sha256_prefix":"3ceb5479bbdec4de","entry":"process","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"GoodJudgeOpen_crawler/request_one_question.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/GoodJudgeOpen_crawler/request_one_question.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ceb5479bbdec4de"}},{"code_sha256_prefix":"c1ccfbc1d9fe2fd7","entry":"replace_llama","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"reorder.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/reorder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1ccfbc1d9fe2fd7"}},{"code_sha256_prefix":"2a20bf01a50913bf","entry":"reorder","repo":"freedomintelligence/freshbench","repo_kind":"official","path":"reorder.py","file_url":"https://github.com/freedomintelligence/freshbench/blob/HEAD/reorder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a20bf01a50913bf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}