{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memory-based-model-editing-at-scale","title":"Memory-Based Model Editing at Scale","arxiv_id":"2206.06520","date":"2022-06-13","proceeding":null,"authors":["Eric Mitchell","Charles Lin","Antoine Bosselut","Christopher D. Manning","Chelsea Finn"],"abstract":"Even the largest neural networks make errors, and once-correct predictions can become invalid as the world changes. Model editors make local updates to the behavior of base (pre-trained) models to inject updated knowledge or correct undesirable behaviors. Existing model editors have shown promise, but also suffer from insufficient expressiveness: they struggle to accurately model an edit's intended scope (examples affected by the edit), leading to inaccurate predictions for test inputs loosely related to the edit, and they often fail altogether after many edits. As a higher-capacity alternative, we propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC), which stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed. To enable more rigorous evaluation of model editors, we introduce three challenging language model editing problems based on question answering, fact-checking, and dialogue generation. We find that only SERAC achieves high performance on all three problems, consistently outperforming existing approaches to model editing by a significant margin. Code, data, and additional project information will be made available at https://sites.google.com/view/serac-editing.","url_abs":"https://arxiv.org/abs/2206.06520v1","url_pdf":"https://arxiv.org/pdf/2206.06520v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memory-based-model-editing-at-scale","repo_url":"https://github.com/eric-mitchell/serac","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"dialogue-generation","task_name":"Dialogue Generation"},{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model-editing","task_name":"Model Editing"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":null,"task_name":"counterfactual"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"serac","method_name":"SERAC"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[{"slug":"serac","name":"SERAC","full_name":"Semi-Parametric Editing with a Retrieval-Augmented Counterfac- tual Model"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.06520","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.06520"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/marcotcr/qa","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eric-mitchell/serac","reach":null}],"summary":{"ran":1,"ran_fixture":2,"ran_draft_wrong":2,"ran_violates":1,"unverified":8},"by_repo_kind":{"listed":{"samples":14,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1946baf07d58cec1","entry":"EditableModel","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1946baf07d58cec1"}},{"code_sha256_prefix":"0048e4b5f0698a9b","entry":"binary_log_probs","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0048e4b5f0698a9b"}},{"code_sha256_prefix":"4930284ea4b67452","entry":"gather_log_probs","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4930284ea4b67452"}},{"code_sha256_prefix":"a559d8be7541e989","entry":"mask_hf_labels","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a559d8be7541e989"}},{"code_sha256_prefix":"e078eeed20f03838","entry":"masked_mean","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e078eeed20f03838"}},{"code_sha256_prefix":"25f478f57e667b2e","entry":"shift_targets","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25f478f57e667b2e"}},{"code_sha256_prefix":"16b1893e8ab378bf","entry":"BertClassifier","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16b1893e8ab378bf"}},{"code_sha256_prefix":"8c62b93f788354a1","entry":"REP","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c62b93f788354a1"}},{"code_sha256_prefix":"8c1523b01e909dd1","entry":"add_padding","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c1523b01e909dd1"}},{"code_sha256_prefix":"cece4bf90fbeb059","entry":"add_sep","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cece4bf90fbeb059"}},{"code_sha256_prefix":"6b3241557810cee9","entry":"es_sentiment","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b3241557810cee9"}},{"code_sha256_prefix":"31acb1a80333fdcf","entry":"masked_log_probs","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31acb1a80333fdcf"}},{"code_sha256_prefix":"93f9333355fbe880","entry":"multiclass_log_probs","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93f9333355fbe880"}},{"code_sha256_prefix":"afecba689fd4060e","entry":"scr","repo":"eric-mitchell/serac","repo_kind":"listed","path":"algs/rep.py","file_url":"https://github.com/eric-mitchell/serac/blob/HEAD/algs/rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afecba689fd4060e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}