{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/confidence-regulation-neurons-in-language","title":"Confidence Regulation Neurons in Language Models","arxiv_id":"2406.16254","date":"2024-06-24","proceeding":null,"authors":["Alessandro Stolfo","Ben Wu","Wes Gurnee","Yonatan Belinkov","Xingyi Song","Mrinmaya Sachan","Neel Nanda"],"abstract":"Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This study investigates two critical components believed to influence this uncertainty: the recently discovered entropy neurons and a new set of components that we term token frequency neurons. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. Our work shows that entropy neurons operate by writing onto an unembedding null space, allowing them to impact the residual stream norm with minimal direct effect on the logits themselves. We observe the presence of entropy neurons across a range of models, up to 7 billion parameters. On the other hand, token frequency neurons, which we discover and describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution. Finally, we present a detailed case study where entropy neurons actively manage confidence in the setting of induction, i.e. detecting and continuing repeated subsequences.","url_abs":"https://arxiv.org/abs/2406.16254v2","url_pdf":"https://arxiv.org/pdf/2406.16254v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"confidence-regulation-neurons-in-language","repo_url":"https://github.com/bpwu1/confidence-regulation-neurons","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.16254","atlas_url":"https://app.syntology.ai/?focus=2406.16254","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16254"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bpwu1/confidence-regulation-neurons","reach":null}],"summary":{"ran_fixture":3,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"249f13b006daf98b","entry":"head_bos_ablation_hook","repo":"bpwu1/confidence-regulation-neurons","repo_kind":"official","path":"ablations/run_and_store_ablation_results.py","file_url":"https://github.com/bpwu1/confidence-regulation-neurons/blob/HEAD/ablations/run_and_store_ablation_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"249f13b006daf98b"}},{"code_sha256_prefix":"d6a8c6abba575ecd","entry":"head_random_ablation_hook","repo":"bpwu1/confidence-regulation-neurons","repo_kind":"official","path":"ablations/run_and_store_ablation_results.py","file_url":"https://github.com/bpwu1/confidence-regulation-neurons/blob/HEAD/ablations/run_and_store_ablation_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6a8c6abba575ecd"}},{"code_sha256_prefix":"ff4da0d05ff39b41","entry":"ln_final_scale_hook","repo":"bpwu1/confidence-regulation-neurons","repo_kind":"official","path":"case_studies/induction.py","file_url":"https://github.com/bpwu1/confidence-regulation-neurons/blob/HEAD/case_studies/induction.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff4da0d05ff39b41"}},{"code_sha256_prefix":"742e6f8abe015402","entry":"neuron_ablation_hook","repo":"bpwu1/confidence-regulation-neurons","repo_kind":"official","path":"ablations/run_and_store_ablation_results.py","file_url":"https://github.com/bpwu1/confidence-regulation-neurons/blob/HEAD/ablations/run_and_store_ablation_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"742e6f8abe015402"}},{"code_sha256_prefix":"9b202bec7b7d50ba","entry":"neurons_clipped_ablation_hook","repo":"bpwu1/confidence-regulation-neurons","repo_kind":"official","path":"case_studies/induction.py","file_url":"https://github.com/bpwu1/confidence-regulation-neurons/blob/HEAD/case_studies/induction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b202bec7b7d50ba"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}