{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-robust-and-cost-efficient-knowledge","title":"Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs","arxiv_id":"2408.06621","date":"2024-08-13","proceeding":null,"authors":["Sungmin Cha","Sungjun Cho","Dasol Hwang","Moontae Lee"],"abstract":"Large Language Models (LLMs) have demonstrated strong reasoning and memorization capabilities via pretraining on massive textual corpora. However, this poses risk of privacy and copyright violations, highlighting the need for efficient machine unlearning methods that remove sensitive data without retraining from scratch. While Gradient Ascent (GA) is commonly used to unlearn by reducing the likelihood of generating unwanted content, it leads to unstable optimization and catastrophic forgetting of retrained knowledge. We find that combining GA with low-rank adaptation results in poor trade-offs between computational cost and generative performance. To address these challenges, we propose Low-rank Knowledge Unlearning (LoKU), a novel framework that enables robust and efficient unlearning for LLMs. First, we introduce Inverted Hinge Loss, which suppresses unwanted tokens while maintaining fluency by boosting the probability of the next most likely token. Second, we develop a data-adaptive initialization for LoRA adapters via low-rank approximation weighted with relative Fisher information, thereby focusing updates on parameters critical for removing targeted knowledge. Experiments on the Training Data Extraction Challenge dataset using GPT-Neo models as well as on the TOFU benchmark with Phi-1.5B and Llama2-7B models demonstrate that our approach effectively removes sensitive information while maintaining reasoning and generative capabilities with minimal impact. Our implementation can be found in https://github.com/csm9493/efficient-llm-unlearning.","url_abs":"https://arxiv.org/abs/2408.06621v4","url_pdf":"https://arxiv.org/pdf/2408.06621v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-robust-and-cost-efficient-knowledge","repo_url":"https://github.com/csm9493/efficient-llm-unlearning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-unlearning","task_name":"Machine Unlearning"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"ga","method_name":"GA"},{"method_slug":"gpt-neo","method_name":"GPT-Neo"},{"method_slug":"tofu","method_name":"Tofu"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.06621","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.06621"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/csm9493/efficient-llm-unlearning","reach":null}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"649fc48067a48529","entry":"find_all_linear_names","repo":"csm9493/efficient-llm-unlearning","repo_kind":"official","path":"TOFU/finetune.py","file_url":"https://github.com/csm9493/efficient-llm-unlearning/blob/HEAD/TOFU/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"649fc48067a48529"}},{"code_sha256_prefix":"7af66f4ab4a74bf7","entry":"find_all_linear_names","repo":"csm9493/efficient-llm-unlearning","repo_kind":"official","path":"TOFU/forget.py","file_url":"https://github.com/csm9493/efficient-llm-unlearning/blob/HEAD/TOFU/forget.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7af66f4ab4a74bf7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}