{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/post-training-sparse-attention-with-double","title":"Post-Training Sparse Attention with Double Sparsity","arxiv_id":"2408.07092","date":"2024-08-11","proceeding":null,"authors":["Shuo Yang","Ying Sheng","Joseph E. Gonzalez","Ion Stoica","Lianmin Zheng"],"abstract":"The inference process for large language models is slow and memory-intensive, with one of the most critical bottlenecks being excessive Key-Value (KV) cache accesses. This paper introduces \"Double Sparsity,\" a novel post-training sparse attention technique designed to alleviate this bottleneck by reducing KV cache access. Double Sparsity combines token sparsity, which focuses on utilizing only the important tokens for computing self-attention, with channel sparsity, an approach that uses important feature channels for identifying important tokens. Our key insight is that the pattern of channel sparsity is relatively static, allowing us to use offline calibration to make it efficient at runtime, thereby enabling accurate and efficient identification of important tokens. Moreover, this method can be combined with offloading to achieve significant memory usage reduction. Experimental results demonstrate that Double Sparsity can achieve $\\frac{1}{16}$ token and channel sparsity with minimal impact on accuracy across various tasks, including wiki-2 perplexity, key-value retrieval, and long context benchmarks with models including Llama-2-7B, Llama-2-70B, and Mixtral-8x7B. It brings up to a 14.1$\\times$ acceleration in attention operations and a 1.9$\\times$ improvement in end-to-end inference on GPUs. With offloading, it achieves a decoding speed acceleration of 16.3$\\times$ compared to state-of-the-art solutions at a sequence length of 256K. Our code is publicly available at https://github.com/andy-yang-1/DoubleSparse.","url_abs":"https://arxiv.org/abs/2408.07092v2","url_pdf":"https://arxiv.org/pdf/2408.07092v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"post-training-sparse-attention-with-double","repo_url":"https://github.com/andy-yang-1/doublesparse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.07092","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.07092"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andy-yang-1/doublesparse","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_honours":2,"ran_draft_wrong":2,"ran_violates":1,"unverified":13},"by_repo_kind":{"official":{"samples":22,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c9d682400f2b2384","entry":"build_chat","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/pred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9d682400f2b2384"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"aba86ffbed27ba65","entry":"find_multiple","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aba86ffbed27ba65"}},{"code_sha256_prefix":"968f7cbd56cd3206","entry":"multinomial_sample_one_no_sync","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/generate.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"968f7cbd56cd3206"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"4b9ff9b601e07731","entry":"parse_args","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/pred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b9ff9b601e07731"}},{"code_sha256_prefix":"4489113b536ca6eb","entry":"post_process","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/pred.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4489113b536ca6eb"}},{"code_sha256_prefix":"6c58a3a856b83402","entry":"pseudo_quantize","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"AIME/modify_llama.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/AIME/modify_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c58a3a856b83402"}},{"code_sha256_prefix":"1102ffb719fa5cde","entry":"apply_rotary_emb","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1102ffb719fa5cde"}},{"code_sha256_prefix":"1bc3eae58a3d0167","entry":"convert_h2o","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/h2o_llama.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/h2o_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bc3eae58a3d0167"}},{"code_sha256_prefix":"423b6b0a2a7a06fb","entry":"convert_kvcache_llama_heavy_recent","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"AIME/modify_llama.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/AIME/modify_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"423b6b0a2a7a06fb"}},{"code_sha256_prefix":"18842172af05afe4","entry":"convert_kvcache_qwen2_heavy_recent","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"AIME/modify_qwen2.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/AIME/modify_qwen2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18842172af05afe4"}},{"code_sha256_prefix":"39e907bab1c70346","entry":"convert_llama_channel_config","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"AIME/modify_llama.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/AIME/modify_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39e907bab1c70346"}},{"code_sha256_prefix":"1ced27a7a4ec46b8","entry":"convert_qwen2_channel_config","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"AIME/modify_qwen2.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/AIME/modify_qwen2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ced27a7a4ec46b8"}},{"code_sha256_prefix":"55ec8fea60e1d7e7","entry":"dynamically_quantize_per_channel","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/quantize.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/quantize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55ec8fea60e1d7e7"}},{"code_sha256_prefix":"dbabfb89a4b530ac","entry":"get_group_qparams","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/quantize.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/quantize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbabfb89a4b530ac"}},{"code_sha256_prefix":"fd0cfe048b12c6f2","entry":"logits_to_probs","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/generate.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/generate.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd0cfe048b12c6f2"}},{"code_sha256_prefix":"146cdcef0c195fd7","entry":"pack_scales_and_zeros","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/quantize.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/quantize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"146cdcef0c195fd7"}},{"code_sha256_prefix":"26cf476e99a51b14","entry":"precompute_freqs_cis","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26cf476e99a51b14"}},{"code_sha256_prefix":"0166832f476ba7ea","entry":"reset_h2o","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"LongBench/h2o_llama.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/LongBench/h2o_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0166832f476ba7ea"}},{"code_sha256_prefix":"b0856105fbfe6f4f","entry":"sample","repo":"andy-yang-1/doublesparse","repo_kind":"official","path":"models/generate.py","file_url":"https://github.com/andy-yang-1/doublesparse/blob/HEAD/models/generate.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0856105fbfe6f4f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}