{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-17367","title":"Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers","arxiv_id":"2601.17367","date":"2026-01-24","proceeding":"ICML","authors":["Zecheng Tang","Quantong Qiu","Yi Yang","Zhiyi Hong","Haiya Xiang","Kebin Liu","Qingqing Dang","Juntao Li","Min Zhang"],"abstract":"The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method.","url_abs":"https://arxiv.org/abs/2601.17367","url_pdf":"https://arxiv.org/pdf/2601.17367","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.17367","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.17367"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/LCM-Lab/Elastic-Attention","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"found_in_text":{"samples":12,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/src/model_utils.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/src/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"583539efd6fd01fb","entry":"apply_rotary_pos_emb","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/src/model_utils.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/src/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"583539efd6fd01fb"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/src/model_utils.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/src/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"2ed34e7143f94ce9","entry":"apply_rotary_emb_torch","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/training/modeling_flash_qwen.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/training/modeling_flash_qwen.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ed34e7143f94ce9"}},{"code_sha256_prefix":"412fcbe6527509d8","entry":"cdf_stretched_concrete","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/training/attention_mask.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/training/attention_mask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"412fcbe6527509d8"}},{"code_sha256_prefix":"4b4f5a76f5aeef0e","entry":"create_causal_mask","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/src/utils.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b4f5a76f5aeef0e"}},{"code_sha256_prefix":"410c0139054f9f9c","entry":"deterministic_z_from_log_alpha","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/training/attention_mask.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/training/attention_mask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"410c0139054f9f9c"}},{"code_sha256_prefix":"1872408025927f3f","entry":"find_blocks_chunked","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/src/utils.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1872408025927f3f"}},{"code_sha256_prefix":"13f436e0309e6fd0","entry":"generate_streaming_info_blocksparse_flash_attn","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/training/eval/modeling_flash_qwen.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/training/eval/modeling_flash_qwen.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13f436e0309e6fd0"}},{"code_sha256_prefix":"3fb9135f1581e90e","entry":"get_mask","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/training/modeling_flash_qwen.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/training/modeling_flash_qwen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3fb9135f1581e90e"}},{"code_sha256_prefix":"07ce21c57ca66278","entry":"sample_z_from_u","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/training/attention_mask.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/training/attention_mask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07ce21c57ca66278"}},{"code_sha256_prefix":"4476c5672f1aafdf","entry":"worker_pack_chunk","repo":"LCM-Lab/Elastic-Attention","repo_kind":"found_in_text","path":"elasticattn/training/dataset_packing.py","file_url":"https://github.com/LCM-Lab/Elastic-Attention/blob/HEAD/elasticattn/training/dataset_packing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4476c5672f1aafdf"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}