{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generalization-v-s-memorization-tracing","title":"Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data","arxiv_id":"2407.14985","date":"2024-07-20","proceeding":null,"authors":["Xinyi Wang","Antonis Antoniades","Yanai Elazar","Alfonso Amayuelas","Alon Albalak","Kexun Zhang","William Yang Wang"],"abstract":"The impressive capabilities of large language models (LLMs) have sparked debate over whether these models genuinely generalize to unseen tasks or predominantly rely on memorizing vast amounts of pretraining data. To explore this issue, we introduce an extended concept of memorization, distributional memorization, which measures the correlation between the LLM output probabilities and the pretraining data frequency. To effectively capture task-specific pretraining data frequency, we propose a novel task-gram language model, which is built by counting the co-occurrence of semantically related $n$-gram pairs from task inputs and outputs in the pretraining corpus. Using the Pythia models trained on the Pile dataset, we evaluate four distinct tasks: machine translation, factual question answering, world knowledge understanding, and math reasoning. Our findings reveal varying levels of memorization, with the strongest effect observed in factual question answering. Furthermore, while model performance improves across all tasks as LLM size increases, only factual question answering shows an increase in memorization, whereas machine translation and reasoning tasks exhibit greater generalization, producing more novel outputs. This study demonstrates that memorization plays a larger role in simpler, knowledge-intensive tasks, while generalization is the key for harder, reasoning-based tasks, providing a scalable method for analyzing large pretraining corpora in greater depth. We also show the practical implications of our analysis through a novel prompt optimization algorithm.","url_abs":"https://arxiv.org/abs/2407.14985v4","url_pdf":"https://arxiv.org/pdf/2407.14985v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"math","task_name":"Math"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"pythia","method_name":"Pythia"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.14985","atlas_url":"https://app.syntology.ai/?focus=2407.14985","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.14985"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/EleutherAI/the-pile","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_honours":1,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":15,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"128c815587eb6f9c","entry":"cont","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/fix_empty_lines.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/fix_empty_lines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"128c815587eb6f9c"}},{"code_sha256_prefix":"400223165519340e","entry":"despace","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/fix_dm_math.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/fix_dm_math.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"400223165519340e"}},{"code_sha256_prefix":"102a3e6041b8359d","entry":"fix","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/fix_dm_math.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/fix_dm_math.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"102a3e6041b8359d"}},{"code_sha256_prefix":"79d53604e4138cba","entry":"id","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"the_pile/utils.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/the_pile/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79d53604e4138cba"}},{"code_sha256_prefix":"0479fb5f08e12e14","entry":"mean","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/lang_len_analysis_pass2.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/lang_len_analysis_pass2.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0479fb5f08e12e14"}},{"code_sha256_prefix":"ea45516e7c59d783","entry":"readf","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/join.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/join.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea45516e7c59d783"}},{"code_sha256_prefix":"2ce3a6653bbabf47","entry":"rewrite_name","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/lang_len_analysis_pass2.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/lang_len_analysis_pass2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ce3a6653bbabf47"}},{"code_sha256_prefix":"2246bc566da383d9","entry":"stableorder","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"the_pile/utils.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/the_pile/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2246bc566da383d9"}},{"code_sha256_prefix":"2f9c8f709caa01bc","entry":"stddev","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/lang_len_analysis_pass2.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/lang_len_analysis_pass2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f9c8f709caa01bc"}},{"code_sha256_prefix":"7ed0726edbb13bb5","entry":"take","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"the_pile/pile.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/the_pile/pile.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ed0726edbb13bb5"}},{"code_sha256_prefix":"66211bd1a2260664","entry":"utf8len","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"the_pile/utils.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/the_pile/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66211bd1a2260664"}},{"code_sha256_prefix":"fc867bf5f0532c70","entry":"analyze","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/lang_len_analysis_pass1.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/lang_len_analysis_pass1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc867bf5f0532c70"}},{"code_sha256_prefix":"5d4f4f3f6f2bb5f0","entry":"json_parser","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"the_pile/tfds_pile.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/the_pile/tfds_pile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d4f4f3f6f2bb5f0"}},{"code_sha256_prefix":"c19e7511b7716d16","entry":"language","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/lang_len_analysis_pass1.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/lang_len_analysis_pass1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c19e7511b7716d16"}},{"code_sha256_prefix":"0ada8308c7e8d45e","entry":"lengths","repo":"EleutherAI/the-pile","repo_kind":"found_in_text","path":"processing_scripts/lang_len_analysis_pass1.py","file_url":"https://github.com/EleutherAI/the-pile/blob/HEAD/processing_scripts/lang_len_analysis_pass1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ada8308c7e8d45e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}