{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-24818","title":"Spiking the training data to correct for test set contamination","arxiv_id":"2605.24818","date":"2026-05-24","proceeding":null,"authors":["Johnny Tian-Zheng Wei","Jerry Li","Ameya Godbole","Robin Jia"],"abstract":"The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our core proposal is to spike the training data by intentionally contaminating some test examples at known rates. The spiked examples can then be used to calibrate predictors of model memorization which enable principled statistical correction of inflated test scores. To evaluate different correction estimators, we first present a simulation framework based on the Hubble models. Hubble models come in minimal pairs, where the perturbed model was deliberately contaminated with several test sets, while the standard model was not, serving as the counterfactual and correction target. We consider estimators that use information from a memorization predictor, correctness predictor, or both. In simulation, we establish basic statistical intuitions and show that estimators leveraging memorization and correctness information are better than naive estimation which makes no correction at all. We then instantiate several memorization and correctness predictors, and find that simple predictors such as Platt-scaled membership inference metrics provide good signal for correction. Finally, we examine the practical considerations of spiking. Simple memorization predictors need no more than 10 examples for calibration and often transfer from one dataset to another. Taken together, spiking is a promising solution for test set contamination.","url_abs":"https://arxiv.org/abs/2605.24818","url_pdf":"https://arxiv.org/pdf/2605.24818","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.24818","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.24818"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Jeli04/spiking-tsc","reach":null}],"summary":{"ran":3,"ran_honours":6,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"a5fe544eb44174fa","entry":"ItemPool","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5fe544eb44174fa"}},{"code_sha256_prefix":"81b224c4d9c88470","entry":"SamplerConfig","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81b224c4d9c88470"}},{"code_sha256_prefix":"acbbe5661ed8bad8","entry":"TestSet","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"acbbe5661ed8bad8"}},{"code_sha256_prefix":"333dc17e419e2ed7","entry":"combined_estimator","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"333dc17e419e2ed7"}},{"code_sha256_prefix":"d1d04939f0491b76","entry":"imputation_estimator","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1d04939f0491b76"}},{"code_sha256_prefix":"6580593af981190a","entry":"ipw_estimator","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6580593af981190a"}},{"code_sha256_prefix":"7ee22197ab18ee00","entry":"naive_estimator","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ee22197ab18ee00"}},{"code_sha256_prefix":"b7c311e3b21006cf","entry":"sample_correlated","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7c311e3b21006cf"}},{"code_sha256_prefix":"c67500ef8cf0204b","entry":"sample_random","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c67500ef8cf0204b"}},{"code_sha256_prefix":"e0a6a333d8eae1f5","entry":"run_sim","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0a6a333d8eae1f5"}},{"code_sha256_prefix":"674fc45b96aedbd9","entry":"sample_test_set","repo":"Jeli04/spiking-tsc","repo_kind":"found_in_text","path":"spiking/adjustment/run_simulation.py","file_url":"https://github.com/Jeli04/spiking-tsc/blob/HEAD/spiking/adjustment/run_simulation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"674fc45b96aedbd9"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}