{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-unreasonable-ineffectiveness-of-nucleus","title":"The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization","arxiv_id":"2408.16345","date":"2024-08-29","proceeding":null,"authors":["Luka Borec","Philipp Sadler","David Schlangen"],"abstract":"This work analyses the text memorization behavior of large language models (LLMs) when subjected to nucleus sampling. Stochastic decoding methods like nucleus sampling are typically applied to overcome issues such as monotonous and repetitive text generation, which are often observed with maximization-based decoding techniques. We hypothesize that nucleus sampling might also reduce the occurrence of memorization patterns, because it could lead to the selection of tokens outside the memorized sequence. To test this hypothesis we create a diagnostic dataset with a known distribution of duplicates that gives us some control over the likelihood of memorization of certain parts of the training data. Our analysis of two GPT-Neo models fine-tuned on this dataset interestingly shows that (i) an increase of the nucleus size reduces memorization only modestly, and (ii) even when models do not engage in \"hard\" memorization -- a verbatim reproduction of training samples -- they may still display \"soft\" memorization whereby they generate outputs that echo the training data but without a complete one-by-one resemblance.","url_abs":"https://arxiv.org/abs/2408.16345v1","url_pdf":"https://arxiv.org/pdf/2408.16345v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-unreasonable-ineffectiveness-of-nucleus","repo_url":"https://github.com/lukaborec/memorization-nucleus-sampling","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"gpt-neo","method_name":"GPT-Neo"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.16345","atlas_url":"https://app.syntology.ai/?focus=2408.16345","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.16345"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lukaborec/memorization-nucleus-sampling","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"36da153ced354d5d","entry":"attention","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/models/transformer.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36da153ced354d5d"}},{"code_sha256_prefix":"e5ab7da14af15648","entry":"check_if_memorized","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/core/running_experiments.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/core/running_experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5ab7da14af15648"}},{"code_sha256_prefix":"a39d815217b8ba86","entry":"check_if_memorized","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/helper_scripts/check_for_coherence.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/helper_scripts/check_for_coherence.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a39d815217b8ba86"}},{"code_sha256_prefix":"89872027cb37ab15","entry":"clones","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/models/transformer.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89872027cb37ab15"}},{"code_sha256_prefix":"991160fd10ff390f","entry":"create_target_paths","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/core/sampling.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/core/sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"991160fd10ff390f"}},{"code_sha256_prefix":"df4dab3d227aecf8","entry":"parse_json_file","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/helper_scripts/check_for_coherence.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/helper_scripts/check_for_coherence.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df4dab3d227aecf8"}},{"code_sha256_prefix":"502b7864ccf325a7","entry":"run_memorization_test","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/helper_scripts/check_for_coherence.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/helper_scripts/check_for_coherence.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"502b7864ccf325a7"}},{"code_sha256_prefix":"9cd02a85b4800229","entry":"subsequent_mask","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/models/transformer.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/models/transformer.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9cd02a85b4800229"}},{"code_sha256_prefix":"d97cfaeef6e40fef","entry":"batched_perplexity","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/core/evaluate.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/core/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d97cfaeef6e40fef"}},{"code_sha256_prefix":"23c893fd630bb704","entry":"tokenize","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/core/running_experiments.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/core/running_experiments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23c893fd630bb704"}},{"code_sha256_prefix":"8310944d881a6af0","entry":"tokenize_inference","repo":"lukaborec/memorization-nucleus-sampling","repo_kind":"official","path":"memorization/core/evaluate.py","file_url":"https://github.com/lukaborec/memorization-nucleus-sampling/blob/HEAD/memorization/core/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8310944d881a6af0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}