{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/graceful-forgetting-in-generative-language","title":"Graceful Forgetting in Generative Language Models","arxiv_id":"2505.19715","date":"2025-05-26","proceeding":null,"authors":["Chunyang Jiang","Chi-Min Chan","Yiyang Cai","Yulong Liu","Wei Xue","Yike Guo"],"abstract":"Recently, the pretrain-finetune paradigm has become a cornerstone in various deep learning areas. While in general the pre-trained model would promote both effectiveness and efficiency of downstream tasks fine-tuning, studies have shown that not all knowledge acquired during pre-training is beneficial. Some of the knowledge may actually bring detrimental effects to the fine-tuning tasks, which is also known as negative transfer. To address this problem, graceful forgetting has emerged as a promising approach. The core principle of graceful forgetting is to enhance the learning plasticity of the target task by selectively discarding irrelevant knowledge. However, this approach remains underexplored in the context of generative language models, and it is often challenging to migrate existing forgetting algorithms to these models due to architecture incompatibility. To bridge this gap, in this paper we propose a novel framework, Learning With Forgetting (LWF), to achieve graceful forgetting in generative language models. With Fisher Information Matrix weighting the intended parameter updates, LWF computes forgetting confidence to evaluate self-generated knowledge regarding the forgetting task, and consequently, knowledge with high confidence is periodically unlearned during fine-tuning. Our experiments demonstrate that, although thoroughly uncovering the mechanisms of knowledge interaction remains challenging in pre-trained language models, applying graceful forgetting can contribute to enhanced fine-tuning performance.","url_abs":"https://arxiv.org/abs/2505.19715v1","url_pdf":"https://arxiv.org/pdf/2505.19715v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.19715","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.19715"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/rubickkcibur/LWF","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"found_in_text":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7a2798b6dec9fdba","entry":"build_model","repo":"rubickkcibur/LWF","repo_kind":"found_in_text","path":"trainLM_tools.py","file_url":"https://github.com/rubickkcibur/LWF/blob/HEAD/trainLM_tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a2798b6dec9fdba"}},{"code_sha256_prefix":"613767452ccfb987","entry":"metric","repo":"rubickkcibur/LWF","repo_kind":"found_in_text","path":"dataset_processor/gsm8k.py","file_url":"https://github.com/rubickkcibur/LWF/blob/HEAD/dataset_processor/gsm8k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"613767452ccfb987"}},{"code_sha256_prefix":"0229ae5a5851afa2","entry":"metric","repo":"rubickkcibur/LWF","repo_kind":"found_in_text","path":"dataset_processor/gsm8k_es.py","file_url":"https://github.com/rubickkcibur/LWF/blob/HEAD/dataset_processor/gsm8k_es.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0229ae5a5851afa2"}},{"code_sha256_prefix":"30c7c57bcdd3cbfd","entry":"metric","repo":"rubickkcibur/LWF","repo_kind":"found_in_text","path":"dataset_processor/gsm8k_it.py","file_url":"https://github.com/rubickkcibur/LWF/blob/HEAD/dataset_processor/gsm8k_it.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30c7c57bcdd3cbfd"}},{"code_sha256_prefix":"a4dfc117d1bf8efc","entry":"metric","repo":"rubickkcibur/LWF","repo_kind":"found_in_text","path":"dataset_processor/gsm8k_tr.py","file_url":"https://github.com/rubickkcibur/LWF/blob/HEAD/dataset_processor/gsm8k_tr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4dfc117d1bf8efc"}},{"code_sha256_prefix":"e2df12ca1a63a0c7","entry":"mix_multi_data","repo":"rubickkcibur/LWF","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/rubickkcibur/LWF/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2df12ca1a63a0c7"}},{"code_sha256_prefix":"3e29e308b7d77163","entry":"qa_preprocess","repo":"rubickkcibur/LWF","repo_kind":"found_in_text","path":"trainLM_tools.py","file_url":"https://github.com/rubickkcibur/LWF/blob/HEAD/trainLM_tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e29e308b7d77163"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}