{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/openunlearning-accelerating-llm-unlearning","title":"OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics","arxiv_id":"2506.12618","date":"2025-06-14","proceeding":null,"authors":["Vineeth Dorna","Anmol Mekala","Wenlong Zhao","Andrew McCallum","Zachary C. Lipton","J. Zico Kolter","Pratyush Maini"],"abstract":"Robust unlearning is crucial for safely deploying large language models (LLMs) in environments where data privacy, model safety, and regulatory compliance must be ensured. Yet the task is inherently challenging, partly due to difficulties in reliably measuring whether unlearning has truly occurred. Moreover, fragmentation in current methodologies and inconsistent evaluation metrics hinder comparative analysis and reproducibility. To unify and accelerate research efforts, we introduce OpenUnlearning, a standardized and extensible framework designed explicitly for benchmarking both LLM unlearning methods and metrics. OpenUnlearning integrates 9 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP) and also enables analyses of forgetting behaviors across 450+ checkpoints we publicly release. Leveraging OpenUnlearning, we propose a novel meta-evaluation benchmark focused specifically on assessing the faithfulness and robustness of evaluation metrics themselves. We also benchmark diverse unlearning methods and provide a comparative analysis against an extensive evaluation suite. Overall, we establish a clear, community-driven pathway toward rigorous development in LLM unlearning research.","url_abs":"https://arxiv.org/abs/2506.12618v1","url_pdf":"https://arxiv.org/pdf/2506.12618v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"openunlearning-accelerating-llm-unlearning","repo_url":"https://github.com/locuslab/open-unlearning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"fragmentation","method_name":"Fragmentation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.12618","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.12618"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/locuslab/open-unlearning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":8},"by_repo_kind":{"official":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6e72fd5056dda43a","entry":"read_json","repo":"locuslab/open-unlearning","repo_kind":"official","path":"community/methods/AltPO/generate.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/community/methods/AltPO/generate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e72fd5056dda43a"}},{"code_sha256_prefix":"56262c8f8ba375b1","entry":"aggregate_to_1D","repo":"locuslab/open-unlearning","repo_kind":"official","path":"src/evals/metrics/utils.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/src/evals/metrics/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56262c8f8ba375b1"}},{"code_sha256_prefix":"f5fa56fd1884242d","entry":"compute_batch_nll","repo":"locuslab/open-unlearning","repo_kind":"official","path":"src/trainer/utils.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/src/trainer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5fa56fd1884242d"}},{"code_sha256_prefix":"ef1eb32faacefd3a","entry":"compute_dpo_loss","repo":"locuslab/open-unlearning","repo_kind":"official","path":"src/trainer/utils.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/src/trainer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef1eb32faacefd3a"}},{"code_sha256_prefix":"0ad1a4c40d0feee4","entry":"compute_kl_divergence","repo":"locuslab/open-unlearning","repo_kind":"official","path":"src/trainer/utils.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/src/trainer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ad1a4c40d0feee4"}},{"code_sha256_prefix":"c49bbf7aa3eb7304","entry":"dict_transpose","repo":"locuslab/open-unlearning","repo_kind":"official","path":"src/evals/metrics/utils.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/src/evals/metrics/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c49bbf7aa3eb7304"}},{"code_sha256_prefix":"78733604aac3e99e","entry":"get_dataset","repo":"locuslab/open-unlearning","repo_kind":"official","path":"community/methods/AltPO/generate.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/community/methods/AltPO/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78733604aac3e99e"}},{"code_sha256_prefix":"52f38c1b3ab76533","entry":"get_forget_quality","repo":"locuslab/open-unlearning","repo_kind":"official","path":"src/evals/metrics/utils.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/src/evals/metrics/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52f38c1b3ab76533"}},{"code_sha256_prefix":"a7b25126cbb0b30e","entry":"get_model","repo":"locuslab/open-unlearning","repo_kind":"official","path":"community/methods/AltPO/generate.py","file_url":"https://github.com/locuslab/open-unlearning/blob/HEAD/community/methods/AltPO/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7b25126cbb0b30e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}