{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/antileak-bench-preventing-data-contamination","title":"AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge","arxiv_id":"2412.13670","date":"2024-12-18","proceeding":null,"authors":["Xiaobao Wu","Liangming Pan","Yuxi Xie","Ruiwen Zhou","Shuai Zhao","Yubo Ma","Mingzhe Du","Rui Mao","Anh Tuan Luu","William Yang Wang"],"abstract":"Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.","url_abs":"https://arxiv.org/abs/2412.13670v1","url_pdf":"https://arxiv.org/pdf/2412.13670v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"antileak-bench-preventing-data-contamination","repo_url":"https://github.com/bobxwu/antileak-bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.13670","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.13670"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bobxwu/antileak-bench","reach":{"status":"ok"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"89e47e37719a8928","entry":"add_handlers","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/logger.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89e47e37719a8928"}},{"code_sha256_prefix":"79f2a98626bab3df","entry":"build_chat","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"LLMs/LLM.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/LLMs/LLM.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79f2a98626bab3df"}},{"code_sha256_prefix":"008d42c071b99ed5","entry":"check_exist","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"building/build_singlehop.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/building/build_singlehop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"008d42c071b99ed5"}},{"code_sha256_prefix":"97086a8793aa00e3","entry":"check_valid_wikipedia","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"building/build_singlehop.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/building/build_singlehop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97086a8793aa00e3"}},{"code_sha256_prefix":"44e0f29827445b11","entry":"chunk_as_size","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44e0f29827445b11"}},{"code_sha256_prefix":"0346247a69a8ec7f","entry":"chunks_by_num","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0346247a69a8ec7f"}},{"code_sha256_prefix":"b3ab014d4fe57f44","entry":"flatten_list","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3ab014d4fe57f44"}},{"code_sha256_prefix":"8fde7564d5daa655","entry":"get_logger","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/logger.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8fde7564d5daa655"}},{"code_sha256_prefix":"7adbb30495f4e96b","entry":"read_yaml","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/file_utils.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7adbb30495f4e96b"}},{"code_sha256_prefix":"596bc848edd49b55","entry":"restructure_as_namespace","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/file_utils.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"596bc848edd49b55"}},{"code_sha256_prefix":"a440c42570f58eaf","entry":"sample_contexts","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"building/build_singlehop.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/building/build_singlehop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a440c42570f58eaf"}},{"code_sha256_prefix":"3bd2859ac20c2950","entry":"update_args","repo":"bobxwu/antileak-bench","repo_kind":"official","path":"utils/file_utils.py","file_url":"https://github.com/bobxwu/antileak-bench/blob/HEAD/utils/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bd2859ac20c2950"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}