{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/align-then-unlearn-embedding-alignment-for","title":"Align-then-Unlearn: Embedding Alignment for LLM Unlearning","arxiv_id":"2506.13181","date":"2025-06-16","proceeding":null,"authors":["Philipp Spohn","Leander Girrbach","Jessica Bader","Zeynep Akata"],"abstract":"As large language models (LLMs) are trained on massive datasets, they have raised significant privacy and ethical concerns due to their potential to inadvertently retain sensitive information. Unlearning seeks to selectively remove specific data from trained models, such as personal information or copyrighted content. Current approaches targeting specific output sequences at the token level often fail to achieve complete forgetting and remain susceptible to prompt rephrasing. We propose Align-then-Unlearn, a novel framework that performs unlearning in the semantic embedding space rather than directly on output tokens. Align-then-Unlearn first augments the LLM with an embedding prediction module trained to anticipate future context representations. Unlearning is then achieved by fine-tuning the model to minimize the similarity between these predicted embeddings and a target embedding that represents the concept to be removed. Initial results show that Align-then-Unlearn effectively removes targeted knowledge with minimal degradation in overall model utility. These findings suggest that embedding-based unlearning offers a promising and robust approach to removing conceptual knowledge. Our code is available at https://github.com/ExplainableML/align-then-unlearn.","url_abs":"https://arxiv.org/abs/2506.13181v1","url_pdf":"https://arxiv.org/pdf/2506.13181v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"align-then-unlearn-embedding-alignment-for","repo_url":"https://github.com/explainableml/align-then-unlearn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.13181","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.13181"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/explainableml/align-then-unlearn","reach":{"status":"ok"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"69cd3aa44c4878e9","entry":"build_adamw_warmup_optimizer","repo":"explainableml/align-then-unlearn","repo_kind":"official","path":"project/optim.py","file_url":"https://github.com/explainableml/align-then-unlearn/blob/HEAD/project/optim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69cd3aa44c4878e9"}},{"code_sha256_prefix":"f516f86cb17161c3","entry":"build_lr_scheduler","repo":"explainableml/align-then-unlearn","repo_kind":"official","path":"project/optim.py","file_url":"https://github.com/explainableml/align-then-unlearn/blob/HEAD/project/optim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f516f86cb17161c3"}},{"code_sha256_prefix":"304f0b2650caa15d","entry":"build_sgd_optimizer","repo":"explainableml/align-then-unlearn","repo_kind":"official","path":"project/optim.py","file_url":"https://github.com/explainableml/align-then-unlearn/blob/HEAD/project/optim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"304f0b2650caa15d"}},{"code_sha256_prefix":"a754de91333ee450","entry":"load_pre_trained_llm","repo":"explainableml/align-then-unlearn","repo_kind":"official","path":"project/pre_trained_model.py","file_url":"https://github.com/explainableml/align-then-unlearn/blob/HEAD/project/pre_trained_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a754de91333ee450"}},{"code_sha256_prefix":"419e2981fc5349d3","entry":"load_pre_trained_text_embedding_model","repo":"explainableml/align-then-unlearn","repo_kind":"official","path":"project/pre_trained_model.py","file_url":"https://github.com/explainableml/align-then-unlearn/blob/HEAD/project/pre_trained_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"419e2981fc5349d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}