{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/restor-knowledge-recovery-through-machine","title":"RESTOR: Knowledge Recovery through Machine Unlearning","arxiv_id":"2411.00204","date":"2024-10-31","proceeding":null,"authors":["Keivan Rezaei","Khyathi Chandu","Soheil Feizi","Yejin Choi","Faeze Brahman","Abhilasha Ravichander"],"abstract":"Large language models trained on web-scale corpora can memorize undesirable datapoints such as incorrect facts, copyrighted content or sensitive data. Recently, many machine unlearning algorithms have been proposed that aim to `erase' these datapoints from trained models -- that is, revert model behavior to be similar to a model that had never been trained on these datapoints. However, evaluating the success of unlearning algorithms remains an open challenge. In this work, we propose the RESTOR framework for machine unlearning, which evaluates the ability of unlearning algorithms to perform targeted data erasure from models, by evaluating the ability of models to forget the knowledge introduced in these data points, while simultaneously recovering the model's knowledge state had it not encountered these datapoints. RESTOR helps uncover several novel insights about popular unlearning algorithms, and the mechanisms through which they operate -- for instance, identifying that some algorithms merely emphasize forgetting, and that localizing unlearning targets can enhance unlearning performance.","url_abs":"https://arxiv.org/abs/2411.00204v2","url_pdf":"https://arxiv.org/pdf/2411.00204v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"restor-knowledge-recovery-through-machine","repo_url":"https://github.com/k1rezaei/restor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"restor-knowledge-recovery-through-machine","repo_url":"https://github.com/mehrdadsaberi/msa_unlearning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-unlearning","task_name":"Machine Unlearning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.00204","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.00204"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mehrdadsaberi/msa_unlearning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/k1rezaei/restor","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"83396ec95592fb23","entry":"load_yaml","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"83396ec95592fb23"}},{"code_sha256_prefix":"fd8d7039bbcb386b","entry":"load_yaml","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"fd8d7039bbcb386b"}},{"code_sha256_prefix":"0e97fe2f81c1f785","entry":"evaluate_judge_output","repo":"k1rezaei/restor","repo_kind":"official","path":"evaluation/chat_gpt_eval/judge.py","file_url":"https://github.com/k1rezaei/restor/blob/HEAD/evaluation/chat_gpt_eval/judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e97fe2f81c1f785"}},{"code_sha256_prefix":"bf748ba496264524","entry":"evaluate_model_next_token_pred","repo":"k1rezaei/restor","repo_kind":"official","path":"evaluation/eval_model_next_token_pred.py","file_url":"https://github.com/k1rezaei/restor/blob/HEAD/evaluation/eval_model_next_token_pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf748ba496264524"}},{"code_sha256_prefix":"81e189b7f7c80d2f","entry":"evaluate_model_on_fact_dataset","repo":"k1rezaei/restor","repo_kind":"official","path":"evaluation/eval_utils.py","file_url":"https://github.com/k1rezaei/restor/blob/HEAD/evaluation/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81e189b7f7c80d2f"}},{"code_sha256_prefix":"1534bf33a80e946f","entry":"evaluate_model_opinion_on_fact_dataset","repo":"k1rezaei/restor","repo_kind":"official","path":"evaluation/mcqa_eval/mcqa_utils.py","file_url":"https://github.com/k1rezaei/restor/blob/HEAD/evaluation/mcqa_eval/mcqa_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1534bf33a80e946f"}},{"code_sha256_prefix":"07cdcb5843a038df","entry":"get_context","repo":"k1rezaei/restor","repo_kind":"official","path":"evaluation/eval_utils.py","file_url":"https://github.com/k1rezaei/restor/blob/HEAD/evaluation/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07cdcb5843a038df"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}