{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/causal-estimation-of-memorisation-profiles","title":"Causal Estimation of Memorisation Profiles","arxiv_id":"2406.04327","date":"2024-06-06","proceeding":null,"authors":["Pietro Lesci","Clara Meister","Thomas Hofmann","Andreas Vlachos","Tiago Pimentel"],"abstract":"Understanding memorisation in language models has practical and societal implications, e.g., studying models' training dynamics or preventing copyright infringements. Prior work defines memorisation as the causal effect of training with an instance on the model's ability to predict that instance. This definition relies on a counterfactual: the ability to observe what would have happened had the model not seen that instance. Existing methods struggle to provide computationally efficient and accurate estimates of this counterfactual. Further, they often estimate memorisation for a model architecture rather than for a specific model instance. This paper fills an important gap in the literature, proposing a new, principled, and efficient method to estimate memorisation based on the difference-in-differences design from econometrics. Using this method, we characterise a model's memorisation profile--its memorisation trends across training--by only observing its behaviour on a small set of instances throughout training. In experiments with the Pythia model suite, we find that memorisation (i) is stronger and more persistent in larger models, (ii) is determined by data order and learning rate, and (iii) has stable trends across model sizes, thus making memorisation in larger models predictable from smaller ones.","url_abs":"https://arxiv.org/abs/2406.04327v1","url_pdf":"https://arxiv.org/pdf/2406.04327v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"causal-estimation-of-memorisation-profiles","repo_url":"https://github.com/pietrolesci/memorisation-profiles","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"econometrics","task_name":"Econometrics"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[{"method_slug":"pythia","method_name":"Pythia"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.04327","atlas_url":"https://app.syntology.ai/?focus=2406.04327","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.04327"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pietrolesci/memorisation-profiles","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6d15dd7c41fd0a5e","entry":"convert_json_to_pandas","repo":"pietrolesci/memorisation-profiles","repo_kind":"official","path":"src/utilities.py","file_url":"https://github.com/pietrolesci/memorisation-profiles/blob/HEAD/src/utilities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d15dd7c41fd0a5e"}},{"code_sha256_prefix":"d2d591b2619db1bd","entry":"flatten","repo":"pietrolesci/memorisation-profiles","repo_kind":"official","path":"src/utilities.py","file_url":"https://github.com/pietrolesci/memorisation-profiles/blob/HEAD/src/utilities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2d591b2619db1bd"}},{"code_sha256_prefix":"a0a0c7a71d2fa750","entry":"ld_to_dl","repo":"pietrolesci/memorisation-profiles","repo_kind":"official","path":"src/utilities.py","file_url":"https://github.com/pietrolesci/memorisation-profiles/blob/HEAD/src/utilities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a0a0c7a71d2fa750"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}