{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/active-dormant-attention-heads","title":"Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs","arxiv_id":"2410.13835","date":"2024-10-17","proceeding":null,"authors":["Tianyu Guo","Druv Pai","Yu Bai","Jiantao Jiao","Michael I. Jordan","Song Mei"],"abstract":"Practitioners have consistently observed three puzzling phenomena in transformer-based large language models (LLMs): attention sinks, value-state drains, and residual-state peaks, collectively referred to as extreme-token phenomena. These phenomena are characterized by certain so-called \"sink tokens\" receiving disproportionately high attention weights, exhibiting significantly smaller value states, and having much larger residual-state norms than those of other tokens. These extreme tokens give rise to various challenges in LLM inference, quantization, and interpretability. We elucidate the mechanisms behind extreme-token phenomena. First, we show that these phenomena arise in very simple architectures -- transformers with one to three layers -- trained on a toy model, the Bigram-Backcopy (BB) task. In this setting, we identify an active-dormant mechanism, where attention heads become sinks for specific input domains while remaining non-sinks for others. Our theoretical analysis of the training dynamics reveals that these phenomena are driven by a mutual reinforcement mechanism. Building on these insights, we propose strategies to mitigate extreme-token phenomena during pretraining, including replacing softmax with ReLU and Adam with SGD. Next, we extend our analysis to pretrained LLMs, including Llama and OLMo, showing that many attention heads exhibit a similar active-dormant mechanism as in the BB task, and that the mutual reinforcement mechanism also governs the emergence of extreme-token phenomena during LLM pretraining. Our results reveal that many of the static and dynamic properties of extreme-token phenomena predicted by the BB task align with observations in pretrained LLMs.","url_abs":"https://arxiv.org/abs/2410.13835v2","url_pdf":"https://arxiv.org/pdf/2410.13835v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"active-dormant-attention-heads","repo_url":"https://github.com/guotianyu2000/active-dormant-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.13835","atlas_url":"https://app.syntology.ai/?focus=2410.13835","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.13835"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/guotianyu2000/active-dormant-attention","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"bf01551b400630d4","entry":"float_to_str","repo":"guotianyu2000/active-dormant-attention","repo_kind":"official","path":"BBM/data.py","file_url":"https://github.com/guotianyu2000/active-dormant-attention/blob/HEAD/BBM/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf01551b400630d4"}},{"code_sha256_prefix":"078d3b529478aba7","entry":"add_training_info","repo":"guotianyu2000/active-dormant-attention","repo_kind":"official","path":"BBM/probe_utils.py","file_url":"https://github.com/guotianyu2000/active-dormant-attention/blob/HEAD/BBM/probe_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"078d3b529478aba7"}},{"code_sha256_prefix":"a8e3f941e9a634be","entry":"get_model_name","repo":"guotianyu2000/active-dormant-attention","repo_kind":"official","path":"BBM/probe_utils.py","file_url":"https://github.com/guotianyu2000/active-dormant-attention/blob/HEAD/BBM/probe_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8e3f941e9a634be"}},{"code_sha256_prefix":"3368d9b6e37f4dca","entry":"make_dataset","repo":"guotianyu2000/active-dormant-attention","repo_kind":"official","path":"BBM/data.py","file_url":"https://github.com/guotianyu2000/active-dormant-attention/blob/HEAD/BBM/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3368d9b6e37f4dca"}},{"code_sha256_prefix":"87bf5b2f48be9d8b","entry":"make_dataset_old","repo":"guotianyu2000/active-dormant-attention","repo_kind":"official","path":"BBM/data.py","file_url":"https://github.com/guotianyu2000/active-dormant-attention/blob/HEAD/BBM/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87bf5b2f48be9d8b"}},{"code_sha256_prefix":"3665aa821f95e5f1","entry":"plot_attns","repo":"guotianyu2000/active-dormant-attention","repo_kind":"official","path":"BBM/probe_utils.py","file_url":"https://github.com/guotianyu2000/active-dormant-attention/blob/HEAD/BBM/probe_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3665aa821f95e5f1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}