{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gated-attention-for-large-language-models-non","title":"Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free","arxiv_id":"2505.06708","date":"2025-05-10","proceeding":null,"authors":["Zihan Qiu","Zekun Wang","Bo Zheng","Zeyu Huang","Kaiyue Wen","Songlin Yang","Rui Men","Le Yu","Fei Huang","Suozhi Huang","Dayiheng Liu","Jingren Zhou","Junyang Lin"],"abstract":"Gating mechanisms have been widely utilized, from early models like LSTMs and Highway Networks to recent state space models, linear attention, and also softmax attention. Yet, existing literature rarely examines the specific effects of gating. In this work, we conduct comprehensive experiments to systematically investigate gating-augmented softmax attention variants. Specifically, we perform a comprehensive comparison over 30 variants of 15B Mixture-of-Experts (MoE) models and 1.7B dense models trained on a 3.5 trillion token dataset. Our central finding is that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance. This modification also enhances training stability, tolerates larger learning rates, and improves scaling properties. By comparing various gating positions and computational variants, we attribute this effectiveness to two key factors: (1) introducing non-linearity upon the low-rank mapping in the softmax attention, and (2) applying query-dependent sparse gating scores to modulate the SDPA output. Notably, we find this sparse gating mechanism mitigates 'attention sink' and enhances long-context extrapolation performance, and we also release related $\\href{https://github.com/qiuzh20/gated_attention}{codes}$ and $\\href{https://huggingface.co/QwQZh/gated_attention}{models}$ to facilitate future research.","url_abs":"https://arxiv.org/abs/2505.06708v1","url_pdf":"https://arxiv.org/pdf/2505.06708v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gated-attention-for-large-language-models-non","repo_url":"https://github.com/qiuzh20/gated_attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"highway-networks","method_name":"Highway networks"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.06708","atlas_url":"https://app.syntology.ai/?focus=2505.06708","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.06708"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qiuzh20/gated_attention","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"qiuzh20/gated_attention","repo_kind":"official","path":"modeling_qwen3.py","file_url":"https://github.com/qiuzh20/gated_attention/blob/HEAD/modeling_qwen3.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"qiuzh20/gated_attention","repo_kind":"official","path":"modeling_qwen3.py","file_url":"https://github.com/qiuzh20/gated_attention/blob/HEAD/modeling_qwen3.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"qiuzh20/gated_attention","repo_kind":"official","path":"modeling_qwen3.py","file_url":"https://github.com/qiuzh20/gated_attention/blob/HEAD/modeling_qwen3.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}