{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/propagating-knowledge-updates-to-lms-through-1","title":"Propagating Knowledge Updates to LMs Through Distillation","arxiv_id":"2306.09306","date":"2023-06-15","proceeding":"NeurIPS 2023 11","authors":["Shankar Padmanabhan","Yasumasa Onoe","Michael J. Q. Zhang","Greg Durrett","Eunsol Choi"],"abstract":"Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters. While prior methods for updating knowledge in LMs successfully inject atomic facts, updated LMs fail to make inferences based on injected facts. In this work, we demonstrate that a context distillation-based approach can both impart knowledge about entities and propagate that knowledge to enable broader inferences. Our approach consists of two stages: transfer set generation and distillation on the transfer set. We first generate a transfer set by prompting a language model to generate continuations from the entity definition. Then, we update the model parameters so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine-tuning and other gradient-based knowledge-editing methods. Moreover, it does not compromise performance in other contexts, even when injecting the definitions of up to 150 entities at once.","url_abs":"https://arxiv.org/abs/2306.09306v2","url_pdf":"https://arxiv.org/pdf/2306.09306v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"propagating-knowledge-updates-to-lms-through-1","repo_url":"https://github.com/shankarp8/knowledge_distillation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"knowledge-editing","task_name":"knowledge editing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.09306","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.09306"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/shankarp8/knowledge_distillation","reach":null}],"summary":{"ran":3,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"d23db44d1a2e99aa","entry":"compute_specificity_ecbd","repo":"shankarp8/knowledge_distillation","repo_kind":"official","path":"src/edit_func.py","file_url":"https://github.com/shankarp8/knowledge_distillation/blob/HEAD/src/edit_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d23db44d1a2e99aa"}},{"code_sha256_prefix":"90fa81c5637fcec8","entry":"compute_specificity_entity_inferences","repo":"shankarp8/knowledge_distillation","repo_kind":"official","path":"src/edit_func.py","file_url":"https://github.com/shankarp8/knowledge_distillation/blob/HEAD/src/edit_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90fa81c5637fcec8"}},{"code_sha256_prefix":"64ab493370e4f4c5","entry":"generate_sample","repo":"shankarp8/knowledge_distillation","repo_kind":"official","path":"src/edit_func.py","file_url":"https://github.com/shankarp8/knowledge_distillation/blob/HEAD/src/edit_func.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64ab493370e4f4c5"}},{"code_sha256_prefix":"c02311c2e53bf28a","entry":"get_log_probs","repo":"shankarp8/knowledge_distillation","repo_kind":"official","path":"src/edit_func.py","file_url":"https://github.com/shankarp8/knowledge_distillation/blob/HEAD/src/edit_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c02311c2e53bf28a"}},{"code_sha256_prefix":"1db31f7a27d2cecf","entry":"ft_distill_gpt","repo":"shankarp8/knowledge_distillation","repo_kind":"official","path":"src/edit_func.py","file_url":"https://github.com/shankarp8/knowledge_distillation/blob/HEAD/src/edit_func.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1db31f7a27d2cecf"}},{"code_sha256_prefix":"36c41cba2ff4aa42","entry":"gpt_distill_after_entity_span","repo":"shankarp8/knowledge_distillation","repo_kind":"official","path":"src/edit_func.py","file_url":"https://github.com/shankarp8/knowledge_distillation/blob/HEAD/src/edit_func.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36c41cba2ff4aa42"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}