{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rwku-benchmarking-real-world-knowledge","title":"RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models","arxiv_id":"2406.10890","date":"2024-06-16","proceeding":null,"authors":["Zhuoran Jin","Pengfei Cao","Chenhao Wang","Zhitao He","Hongbang Yuan","Jiachun Li","Yubo Chen","Kang Liu","Jun Zhao"],"abstract":"Large language models (LLMs) inevitably memorize sensitive, copyrighted, and harmful knowledge from the training corpus; therefore, it is crucial to erase this knowledge from the models. Machine unlearning is a promising solution for efficiently removing specific knowledge by post hoc modifying models. In this paper, we propose a Real-World Knowledge Unlearning benchmark (RWKU) for LLM unlearning. RWKU is designed based on the following three key factors: (1) For the task setting, we consider a more practical and challenging unlearning setting, where neither the forget corpus nor the retain corpus is accessible. (2) For the knowledge source, we choose 200 real-world famous people as the unlearning targets and show that such popular knowledge is widely present in various LLMs. (3) For the evaluation framework, we design the forget set and the retain set to evaluate the model's capabilities across various real-world applications. Regarding the forget set, we provide four four membership inference attack (MIA) methods and nine kinds of adversarial attack probes to rigorously test unlearning efficacy. Regarding the retain set, we assess locality and utility in terms of neighbor perturbation, general ability, reasoning ability, truthfulness, factuality, and fluency. We conduct extensive experiments across two unlearning scenarios, two models and six baseline methods and obtain some meaningful findings. We release our benchmark and code publicly at http://rwku-bench.github.io for future work.","url_abs":"https://arxiv.org/abs/2406.10890v1","url_pdf":"https://arxiv.org/pdf/2406.10890v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rwku-benchmarking-real-world-knowledge","repo_url":"https://github.com/jinzhuoran/rwku","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"inference-attack","task_name":"Inference Attack"},{"task_slug":"machine-unlearning","task_name":"Machine Unlearning"},{"task_slug":"membership-inference-attack","task_name":"Membership Inference Attack"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"hoc","method_name":"HOC"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.10890","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.10890"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jinzhuoran/rwku","reach":null}],"summary":{"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"7102598353f91881","entry":"get_next_word_predictions","repo":"jinzhuoran/rwku","repo_kind":"official","path":"generation/inference.py","file_url":"https://github.com/jinzhuoran/rwku/blob/HEAD/generation/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7102598353f91881"}},{"code_sha256_prefix":"d029fa02758da921","entry":"score_completions","repo":"jinzhuoran/rwku","repo_kind":"official","path":"generation/inference.py","file_url":"https://github.com/jinzhuoran/rwku/blob/HEAD/generation/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d029fa02758da921"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}