{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-it-possible-to-edit-large-language-models","title":"On the Robustness of Editing Large Language Models","arxiv_id":"2402.05827","date":"2024-02-08","proceeding":null,"authors":["Xinbei Ma","Tianjie Ju","Jiyang Qiu","Zhuosheng Zhang","Hai Zhao","Lifeng Liu","Yulong Wang"],"abstract":"Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates. Model editing enables the manipulation of specific knowledge memories and the behavior of language generation without retraining. However, the robustness of model editing remains an open question. This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI. We focus on three key research questions. RQ1: Can edited LLMs behave consistently resembling communicative AI in realistic situations? RQ2: To what extent does the rephrasing of prompts lead LLMs to deviate from the edited knowledge memory? RQ3: Which knowledge features are correlated with the performance and robustness of editing? Our empirical studies uncover a substantial disparity between existing editing methods and the practical application of LLMs. On rephrased prompts that are flexible but common in realistic applications, the performance of editing experiences a significant decline. Further analysis shows that more popular knowledge is memorized better, easier to recall, and more challenging to edit effectively. Code is publicly available at https://github.com/xbmxb/edit_analysis .","url_abs":"https://arxiv.org/abs/2402.05827v2","url_pdf":"https://arxiv.org/pdf/2402.05827v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-it-possible-to-edit-large-language-models","repo_url":"https://github.com/xbmxb/edit_analysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"model-editing","task_name":"Model Editing"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.05827","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05827"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xbmxb/edit_analysis","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_draft_wrong":3,"unverified":7},"by_repo_kind":{"official":{"samples":16,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"5168a172bbb363a5","entry":"convert_pytorch_state_dict_to_flax","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5168a172bbb363a5"}},{"code_sha256_prefix":"dd8f7751ef859581","entry":"infer_metric_tags_from_eval_results","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modelcard.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd8f7751ef859581"}},{"code_sha256_prefix":"a4491b45d19b7a39","entry":"parse_keras_history","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modelcard.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4491b45d19b7a39"}},{"code_sha256_prefix":"a766a95b9fd5ee9c","entry":"prepare_ctrl_input","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/post_test_fastchat.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/post_test_fastchat.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a766a95b9fd5ee9c"}},{"code_sha256_prefix":"b7bf53f6b6b11a4b","entry":"prepare_xlm_input","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/post_test_fastchat.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/post_test_fastchat.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7bf53f6b6b11a4b"}},{"code_sha256_prefix":"80844fa7cfa6e8d8","entry":"prepare_xlnet_input","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/post_test_fastchat.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/post_test_fastchat.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80844fa7cfa6e8d8"}},{"code_sha256_prefix":"1245856df6f338d3","entry":"profiling","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/baseline_loced.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/baseline_loced.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1245856df6f338d3"}},{"code_sha256_prefix":"a66d2b78670c85e1","entry":"quick_gelu","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modeling_flax_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a66d2b78670c85e1"}},{"code_sha256_prefix":"61779b0406a59cfc","entry":"rename_key_and_reshape_tensor","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61779b0406a59cfc"}},{"code_sha256_prefix":"f4075cdecbb730c5","entry":"check_imports","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/dynamic_module_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f4075cdecbb730c5"}},{"code_sha256_prefix":"9d98806acb3144b7","entry":"dtype_byte_size","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modeling_flax_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d98806acb3144b7"}},{"code_sha256_prefix":"e3ce70ca9258800e","entry":"flax_shard_checkpoint","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modeling_flax_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3ce70ca9258800e"}},{"code_sha256_prefix":"af4d8c865cc2df7b","entry":"get_conv_template","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/conversation.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/conversation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af4d8c865cc2df7b"}},{"code_sha256_prefix":"00cc853a73f3aeb6","entry":"get_relative_import_files","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/dynamic_module_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00cc853a73f3aeb6"}},{"code_sha256_prefix":"450aeffe7f80f917","entry":"get_relative_imports","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/dynamic_module_utils.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"450aeffe7f80f917"}},{"code_sha256_prefix":"ec7b1528a936234d","entry":"is_hf_dataset","repo":"xbmxb/edit_analysis","repo_kind":"official","path":"code/transformers/modelcard.py","file_url":"https://github.com/xbmxb/edit_analysis/blob/HEAD/code/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec7b1528a936234d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}