{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/meow-memory-supervised-llm-unlearning-via","title":"MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts","arxiv_id":"2409.11844","date":"2024-09-18","proceeding":null,"authors":["Tianle Gu","Kexin Huang","Ruilin Luo","Yuanqi Yao","Yujiu Yang","Yan Teng","Yingchun Wang"],"abstract":"Large Language Models (LLMs) can memorize sensitive information, raising concerns about potential misuse. LLM Unlearning, a post-hoc approach to remove this information from trained LLMs, offers a promising solution to mitigate these risks. However, previous practices face three key challenges: 1. Utility: successful unlearning often causes catastrophic collapse on unrelated tasks. 2. Efficiency: many methods either involve adding similarly sized models, which slows down unlearning or inference, or require retain data that are difficult to obtain. 3. Robustness: even effective methods may still leak data via extraction techniques. To address these challenges, we propose MEOW, a simple yet effective gradient descent-based unlearning method. Specifically, we use an offline LLM to generate a set of inverted facts. Then, we design a new metric, MEMO, to quantify memorization in LLMs. Finally, based on the signals provided by MEMO, we select the most appropriate set of inverted facts and finetune the model based on them. We evaluate MEOW on the commonly used unlearn benchmark, ToFU, with Llama2-7B-Chat and Phi-1.5B, and test it on both NLU and NLG tasks. Results demonstrate significant improvement of MEOW in forget quality without substantial loss in model utility. Meanwhile, MEOW does not exhibit significant degradation in NLU or NLG capabilities, and there is even a slight improvement in NLU performance.","url_abs":"https://arxiv.org/abs/2409.11844v1","url_pdf":"https://arxiv.org/pdf/2409.11844v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"meow-memory-supervised-llm-unlearning-via","repo_url":"https://github.com/carol-gutianle/meow","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"memorization","task_name":"Memorization"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.11844","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.11844"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/carol-gutianle/meow","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"5985b70afc6ef2b0","entry":"custom_data_collator","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/dataset.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5985b70afc6ef2b0"}},{"code_sha256_prefix":"b3dab92b14270b57","entry":"custom_data_collator_forget","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/dataset.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3dab92b14270b57"}},{"code_sha256_prefix":"0978589be4224d97","entry":"custom_data_collator_with_indices","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/dataset.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0978589be4224d97"}},{"code_sha256_prefix":"649fc48067a48529","entry":"find_all_linear_names","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/tofu/finetune.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/tofu/finetune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"649fc48067a48529"}},{"code_sha256_prefix":"442bc919060e0f2b","entry":"get_batch_loss","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/utils.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"442bc919060e0f2b"}},{"code_sha256_prefix":"c55fe02db3bba2fc","entry":"get_forget_quality","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/tofu/aggregate.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/tofu/aggregate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c55fe02db3bba2fc"}},{"code_sha256_prefix":"b89fc6709df8d03f","entry":"load_data_from_json","repo":"carol-gutianle/meow","repo_kind":"official","path":"memo/memo.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/memo/memo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b89fc6709df8d03f"}},{"code_sha256_prefix":"956672833b0ef856","entry":"main","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/tofu/aggregate.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/tofu/aggregate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"956672833b0ef856"}},{"code_sha256_prefix":"644003af682c4c68","entry":"get_model_identifiers_from_yaml","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/utils.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"644003af682c4c68"}},{"code_sha256_prefix":"a29a4722ab1dd2d8","entry":"get_model_utility","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/utils.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a29a4722ab1dd2d8"}},{"code_sha256_prefix":"fc4739c113213ace","entry":"get_model_utility","repo":"carol-gutianle/meow","repo_kind":"official","path":"meow/tofu/aggregate.py","file_url":"https://github.com/carol-gutianle/meow/blob/HEAD/meow/tofu/aggregate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc4739c113213ace"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}