{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-does-the-knowledge-neuron-thesis-have-to","title":"What does the Knowledge Neuron Thesis Have to do with Knowledge?","arxiv_id":"2405.02421","date":"2024-05-03","proceeding":null,"authors":["Jingcheng Niu","Andrew Liu","Zining Zhu","Gerald Penn"],"abstract":"We reassess the Knowledge Neuron (KN) Thesis: an interpretation of the mechanism underlying the ability of large language models to recall facts from a training corpus. This nascent thesis proposes that facts are recalled from the training corpus through the MLP weights in a manner resembling key-value memory, implying in effect that \"knowledge\" is stored in the network. Furthermore, by modifying the MLP modules, one can control the language model's generation of factual information. The plausibility of the KN thesis has been demonstrated by the success of KN-inspired model editing methods (Dai et al., 2022; Meng et al., 2022). We find that this thesis is, at best, an oversimplification. Not only have we found that we can edit the expression of certain linguistic phenomena using the same model editing methods but, through a more comprehensive evaluation, we have found that the KN thesis does not adequately explain the process of factual expression. While it is possible to argue that the MLP weights store complex patterns that are interpretable both syntactically and semantically, these patterns do not constitute \"knowledge.\" To gain a more comprehensive understanding of the knowledge representation process, we must look beyond the MLP weights and explore recent models' complex layer structures and attention mechanisms.","url_abs":"https://arxiv.org/abs/2405.02421v1","url_pdf":"https://arxiv.org/pdf/2405.02421v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-does-the-knowledge-neuron-thesis-have-to","repo_url":"https://github.com/frankniujc/kn_thesis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model-editing","task_name":"Model Editing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.02421","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.02421"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/frankniujc/kn_thesis","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":14,"ran_draft_wrong":2,"ran_fixture":1,"unverified":6},"by_repo_kind":{"official":{"samples":23,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b52190ed343f222d","entry":"anaphor_gender_agreement_func","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/blimp.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/blimp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b52190ed343f222d"}},{"code_sha256_prefix":"f7bb8a394624ac94","entry":"determiner_noun_agreement_2_func","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/blimp.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/blimp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7bb8a394624ac94"}},{"code_sha256_prefix":"374edd244d8abc18","entry":"detok","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/blimp.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/blimp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"374edd244d8abc18"}},{"code_sha256_prefix":"dfe7a59e7269bd44","entry":"find_fact_lookup_idx","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/rome/compute_v.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/rome/compute_v.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfe7a59e7269bd44"}},{"code_sha256_prefix":"4a0aa407f834f3d9","entry":"fix_p_neuron","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/knowledge_neurons.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/knowledge_neurons.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a0aa407f834f3d9"}},{"code_sha256_prefix":"44b3ed63235aa487","entry":"get_attributes","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/patch.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/patch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44b3ed63235aa487"}},{"code_sha256_prefix":"98b536c6d2cf8aa6","entry":"get_ff_layer","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/patch.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/patch.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98b536c6d2cf8aa6"}},{"code_sha256_prefix":"d265047cc27fa57f","entry":"get_inv_cov","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/rome/compute_u.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/rome/compute_u.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d265047cc27fa57f"}},{"code_sha256_prefix":"d7207b7f0d4ca645","entry":"get_refined_neurons","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/knowledge_neurons.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/knowledge_neurons.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7207b7f0d4ca645"}},{"code_sha256_prefix":"920b394e7ad80c53","entry":"hierarchical_subsequence","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/util/nethook.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/util/nethook.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"920b394e7ad80c53"}},{"code_sha256_prefix":"bd095a141719ae5d","entry":"plot","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/plot.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd095a141719ae5d"}},{"code_sha256_prefix":"70f6ab8bde55420e","entry":"recursive_copy","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/util/nethook.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/util/nethook.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70f6ab8bde55420e"}},{"code_sha256_prefix":"c10dba9c17ddf3cd","entry":"register_hook","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/patch.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/patch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c10dba9c17ddf3cd"}},{"code_sha256_prefix":"2b2c1c3cf73fd88d","entry":"run_kn_alg","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/knowledge_neurons.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/knowledge_neurons.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b2c1c3cf73fd88d"}},{"code_sha256_prefix":"5db4cdd5b51f6ed4","entry":"slice_list","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/evaluate.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5db4cdd5b51f6ed4"}},{"code_sha256_prefix":"440ff98c2b1ae1aa","entry":"subsequence","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/util/nethook.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/util/nethook.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"440ff98c2b1ae1aa"}},{"code_sha256_prefix":"407c255c5fd63c76","entry":"to_percent","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/plot.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"407c255c5fd63c76"}},{"code_sha256_prefix":"94ccd51944d2fec4","entry":"generate_symmetry","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/data.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94ccd51944d2fec4"}},{"code_sha256_prefix":"7721312b15ad67eb","entry":"generate_synonym","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/data.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7721312b15ad67eb"}},{"code_sha256_prefix":"3b97843e6bca23a9","entry":"get_module_input_output_at_word","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/rome/compute_v.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/rome/compute_v.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b97843e6bca23a9"}},{"code_sha256_prefix":"3df6877291666386","entry":"load_pararel","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/data.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3df6877291666386"}},{"code_sha256_prefix":"7f798a12b14dbf4c","entry":"pre_post_edit_probs","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/plot.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f798a12b14dbf4c"}},{"code_sha256_prefix":"15ba6a8654411eae","entry":"test_prediction_acc","repo":"frankniujc/kn_thesis","repo_kind":"official","path":"kn_code/evaluate.py","file_url":"https://github.com/frankniujc/kn_thesis/blob/HEAD/kn_code/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15ba6a8654411eae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}