{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/avoiding-data-contamination-in-language-model","title":"LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction","arxiv_id":"2312.12343","date":"2023-12-19","proceeding":null,"authors":["Yucheng Li","Frank Guerin","Chenghua Lin"],"abstract":"Data contamination in evaluation is getting increasingly prevalent with the emergence of language models pre-trained on super large, automatically crawled corpora. This problem leads to significant challenges in the accurate assessment of model capabilities and generalisations. In this paper, we propose LatestEval, an automatic method that leverages the most recent texts to create uncontaminated reading comprehension evaluations. LatestEval avoids data contamination by only using texts published within a recent time window, ensuring no overlap with the training corpora of pre-trained language models. We develop the LatestEval automated pipeline to 1) gather the latest texts; 2) identify key information, and 3) construct questions targeting the information while removing the existing answers from the context. This encourages models to infer the answers themselves based on the remaining context, rather than just copy-paste. Our experiments demonstrate that language models exhibit negligible memorisation behaviours on LatestEval as opposed to previous benchmarks, suggesting a significantly reduced risk of data contamination and leading to a more robust evaluation. Data and code are publicly available at: https://github.com/liyucheng09/LatestEval.","url_abs":"https://arxiv.org/abs/2312.12343v3","url_pdf":"https://arxiv.org/pdf/2312.12343v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"avoiding-data-contamination-in-language-model","repo_url":"https://github.com/liyucheng09/latesteval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-model-evaluation","task_name":"Language Model Evaluation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.12343","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.12343"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liyucheng09/latesteval","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"5ccddaa25f6c958c","entry":"compute_pairwise_win_fraction","repo":"liyucheng09/latesteval","repo_kind":"official","path":"figs/winrate.py","file_url":"https://github.com/liyucheng09/latesteval/blob/HEAD/figs/winrate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ccddaa25f6c958c"}},{"code_sha256_prefix":"c0f1a2bb2d2fbc51","entry":"post_cleanup","repo":"liyucheng09/latesteval","repo_kind":"official","path":"bbc_downloader.py","file_url":"https://github.com/liyucheng09/latesteval/blob/HEAD/bbc_downloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0f1a2bb2d2fbc51"}},{"code_sha256_prefix":"d1a4f006302df3d4","entry":"calculate_best_node","repo":"liyucheng09/latesteval","repo_kind":"official","path":"bbc_downloader.py","file_url":"https://github.com/liyucheng09/latesteval/blob/HEAD/bbc_downloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1a4f006302df3d4"}},{"code_sha256_prefix":"de4304969afd0ba3","entry":"fetch_recent_changes","repo":"liyucheng09/latesteval","repo_kind":"official","path":"wikitext_downloader.py","file_url":"https://github.com/liyucheng09/latesteval/blob/HEAD/wikitext_downloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de4304969afd0ba3"}},{"code_sha256_prefix":"c6973a6f774e2e73","entry":"generate_continuation","repo":"liyucheng09/latesteval","repo_kind":"official","path":"eval/contamination.py","file_url":"https://github.com/liyucheng09/latesteval/blob/HEAD/eval/contamination.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6973a6f774e2e73"}},{"code_sha256_prefix":"7072fd7e0b193209","entry":"gpt","repo":"liyucheng09/latesteval","repo_kind":"official","path":"eval/contamination.py","file_url":"https://github.com/liyucheng09/latesteval/blob/HEAD/eval/contamination.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7072fd7e0b193209"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}