{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/invariance-makes-llm-unlearning-resilient","title":"Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning","arxiv_id":"2506.01339","date":"2025-06-02","proceeding":null,"authors":["Changsheng Wang","Yihua Zhang","Jinghan Jia","Parikshit Ram","Dennis Wei","Yuguang Yao","Soumyadeep Pal","Nathalie Baracaldo","Sijia Liu"],"abstract":"Machine unlearning offers a promising solution to privacy and safety concerns in large language models (LLMs) by selectively removing targeted knowledge while preserving utility. However, current methods are highly sensitive to downstream fine-tuning, which can quickly recover forgotten information-even from unrelated tasks. To address this, we introduce invariance into unlearning for the first time, inspired by invariant risk minimization (IRM). Building on this principle, we propose invariant LLM unlearning (ILU), a regularization-based framework that enhances robustness. Notably, ILU generalizes well to diverse fine-tuning tasks, even when trained using a single dataset. A task vector analysis is also provided to further elucidate the rationale behind ILU's effectiveness. Extensive experiments on the WMDP and MUSE benchmark, reveal that ILU significantly outperforms state-of-the-art unlearning methods, including negative preference optimization (NPO) and representation misdirection for unlearning (RMU). Notably, ILU achieves superior unlearning robustness across diverse downstream fine-tuning scenarios (e.g., math, paraphrase detection, and sentiment analysis) while preserving the fine-tuning performance.","url_abs":"https://arxiv.org/abs/2506.01339v1","url_pdf":"https://arxiv.org/pdf/2506.01339v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"invariance-makes-llm-unlearning-resilient","repo_url":"https://github.com/optml-group/unlearn-ilu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-unlearning","task_name":"Machine Unlearning"},{"task_slug":"math","task_name":"Math"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.01339","atlas_url":"https://app.syntology.ai/?focus=2506.01339","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.01339"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OPTML-Group/Unlearn-ILU","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/optml-group/unlearn-ilu","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"88a4f13abd54a00c","entry":"compute_metrics","repo":"OPTML-Group/Unlearn-ILU","repo_kind":"official","path":"src/unlearn/base.py","file_url":"https://github.com/OPTML-Group/Unlearn-ILU/blob/HEAD/src/unlearn/base.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88a4f13abd54a00c"}},{"code_sha256_prefix":"442bc919060e0f2b","entry":"get_batch_loss","repo":"OPTML-Group/Unlearn-ILU","repo_kind":"official","path":"src/unlearn/utils.py","file_url":"https://github.com/OPTML-Group/Unlearn-ILU/blob/HEAD/src/unlearn/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"442bc919060e0f2b"}},{"code_sha256_prefix":"650d5f6bfcd7d9bb","entry":"get_loss","repo":"OPTML-Group/Unlearn-ILU","repo_kind":"official","path":"src/unlearn/base.py","file_url":"https://github.com/OPTML-Group/Unlearn-ILU/blob/HEAD/src/unlearn/base.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"650d5f6bfcd7d9bb"}},{"code_sha256_prefix":"30e138b18347fbb4","entry":"kl_loss","repo":"OPTML-Group/Unlearn-ILU","repo_kind":"official","path":"src/unlearn/KL.py","file_url":"https://github.com/OPTML-Group/Unlearn-ILU/blob/HEAD/src/unlearn/KL.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30e138b18347fbb4"}},{"code_sha256_prefix":"5a2f4fa433151210","entry":"run_generation","repo":"OPTML-Group/Unlearn-ILU","repo_kind":"official","path":"src/unlearn/utils.py","file_url":"https://github.com/OPTML-Group/Unlearn-ILU/blob/HEAD/src/unlearn/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a2f4fa433151210"}},{"code_sha256_prefix":"21b851de3f5356c4","entry":"unlearncollector","repo":"OPTML-Group/Unlearn-ILU","repo_kind":"official","path":"src/dataset/Base.py","file_url":"https://github.com/OPTML-Group/Unlearn-ILU/blob/HEAD/src/dataset/Base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21b851de3f5356c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}