{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/curse-of-high-dimensionality-issue-in","title":"Curse of High Dimensionality Issue in Transformer for Long-context Modeling","arxiv_id":"2505.22107","date":"2025-05-28","proceeding":null,"authors":["Shuhai Zhang","Zeng You","Yaofo Chen","Zhiquan Wen","Qianyue Wang","Zhijie Qiu","Yuanqing Li","Mingkui Tan"],"abstract":"Transformer-based large language models (LLMs) excel in natural language processing tasks by capturing long-range dependencies through self-attention mechanisms. However, long-context modeling faces significant computational inefficiencies due to \\textit{redundant} attention computations: while attention weights are often \\textit{sparse}, all tokens consume \\textit{equal} computational resources. In this paper, we reformulate traditional probabilistic sequence modeling as a \\textit{supervised learning task}, enabling the separation of relevant and irrelevant tokens and providing a clearer understanding of redundancy. Based on this reformulation, we theoretically analyze attention sparsity, revealing that only a few tokens significantly contribute to predictions. Building on this, we formulate attention optimization as a linear coding problem and propose a \\textit{group coding strategy}, theoretically showing its ability to improve robustness against random noise and enhance learning efficiency. Motivated by this, we propose \\textit{Dynamic Group Attention} (DGA), which leverages the group coding to explicitly reduce redundancy by aggregating less important tokens during attention computation. Empirical results show that our DGA significantly reduces computational costs while maintaining competitive performance.Code is available at https://github.com/bolixinyu/DynamicGroupAttention.","url_abs":"https://arxiv.org/abs/2505.22107v3","url_pdf":"https://arxiv.org/pdf/2505.22107v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"curse-of-high-dimensionality-issue-in","repo_url":"https://github.com/bolixinyu/dynamicgroupattention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2505.22107","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.22107"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bolixinyu/dynamicgroupattention","reach":null}],"summary":{"ran_fixture":4,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"39c3b2313c16cdb5","entry":"_weight_pooling","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39c3b2313c16cdb5"}},{"code_sha256_prefix":"95c4a378cd07db2a","entry":"fast_calculate_fusion_weights","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95c4a378cd07db2a"}},{"code_sha256_prefix":"2b175d1d753da1d7","entry":"generate_mask","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b175d1d753da1d7"}},{"code_sha256_prefix":"4c5015108b289c46","entry":"get_primal_indices","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c5015108b289c46"}},{"code_sha256_prefix":"503b031480c7967e","entry":"reposition","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"503b031480c7967e"}},{"code_sha256_prefix":"4dffdfd4a02bca37","entry":"_attention_impl","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4dffdfd4a02bca37"}},{"code_sha256_prefix":"b5acf5baf7263ff1","entry":"attn_fwd","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5acf5baf7263ff1"}},{"code_sha256_prefix":"27840eacf4387429","entry":"dynamic_grouping_attn_forward","repo":"bolixinyu/dynamicgroupattention","repo_kind":"official","path":"codebase/core/replace_llama_attn.py","file_url":"https://github.com/bolixinyu/dynamicgroupattention/blob/HEAD/codebase/core/replace_llama_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27840eacf4387429"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}