{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/squeezed-attention-accelerating-long-context","title":"Squeezed Attention: Accelerating Long Context Length LLM Inference","arxiv_id":"2411.09688","date":"2024-11-14","proceeding":null,"authors":["Coleman Hooper","Sehoon Kim","Hiva Mohammadzadeh","Monishwaran Maheswaran","June Paik","Michael W. Mahoney","Kurt Keutzer","Amir Gholami"],"abstract":"Emerging Large Language Model (LLM) applications require long input prompts to perform complex downstream tasks like document analysis and code generation. For these long context length applications, the length of the input prompt poses a significant challenge in terms of inference efficiency since the inference costs increase linearly with sequence length. However, for many of these applications, much of the context in the prompt is fixed across different user inputs, thereby providing the opportunity to perform offline optimizations to process user inputs quickly, as they are received. In this work, we propose Squeezed Attention as a mechanism to accelerate LLM applications where a large portion of the input prompt is fixed. We first leverage K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. During inference, we compare query tokens from the user input with the centroids to predict which of the keys from the fixed context are semantically relevant and need to be loaded during inference. We then compute exact attention using only these important keys from the fixed context, thereby reducing bandwidth and computational costs. We also extend our method to use a hierarchical centroid lookup to identify important keys, which can reduce the complexity of attention from linear to logarithmic with respect to the context length. We implement optimized Triton kernels for centroid comparison and sparse FlashAttention with important keys, achieving more than 4x speedups during both the prefill and generation phases for long-context inference. Furthermore, we have extensively evaluated our method on various long-context benchmarks including LongBench, where it achieves a 3x reduction in KV cache budget without accuracy loss and up to an 8x reduction with <0.5 point accuracy gap for various models.","url_abs":"https://arxiv.org/abs/2411.09688v2","url_pdf":"https://arxiv.org/pdf/2411.09688v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"squeezed-attention-accelerating-long-context","repo_url":"https://github.com/SqueezeAILab/SqueezedAttention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"k-means-clustering","method_name":"k-Means Clustering"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.09688","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.09688"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SqueezeAILab/SqueezedAttention","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"unverified":11},"by_repo_kind":{"official":{"samples":15,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"a9e7f2cdf016b88b","entry":"find_layers","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"utils/modelutils.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/utils/modelutils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9e7f2cdf016b88b"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"66d002e5d75a1496","entry":"build_chat","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"squeezedattention/utils.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/squeezedattention/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66d002e5d75a1496"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"b65e0b634a94c717","entry":"get_module_names","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"utils/model_parse.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/utils/model_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b65e0b634a94c717"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"b66c43d90a1a3a1b","entry":"load_model","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"utils/model_parse.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/utils/model_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b66c43d90a1a3a1b"}},{"code_sha256_prefix":"71856fc92741e6f7","entry":"load_model_and_tokenizer","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/LongBench/pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71856fc92741e6f7"}},{"code_sha256_prefix":"8c953c2e0af15854","entry":"parse_args","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/LongBench/pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c953c2e0af15854"}},{"code_sha256_prefix":"dd982ece7960b018","entry":"parse_model","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"utils/model_parse.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/utils/model_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd982ece7960b018"}},{"code_sha256_prefix":"a521d80a7ee5e568","entry":"run_global_threshold","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"squeezedattention/clustering.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/squeezedattention/clustering.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a521d80a7ee5e568"}},{"code_sha256_prefix":"983ad6de59ae59a1","entry":"truncate_fn","repo":"SqueezeAILab/SqueezedAttention","repo_kind":"official","path":"squeezedattention/utils.py","file_url":"https://github.com/SqueezeAILab/SqueezedAttention/blob/HEAD/squeezedattention/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"983ad6de59ae59a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}