{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sageattention-accurate-8-bit-attention-for","title":"SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration","arxiv_id":"2410.02367","date":"2024-10-03","proceeding":null,"authors":["Jintao Zhang","Jia Wei","Haofeng Huang","Pengle Zhang","Jun Zhu","Jianfei Chen"],"abstract":"The transformer architecture predominates across various models. As the heart of the transformer, attention has a computational complexity of O(N^2), compared to O(N) for linear transformations. When handling large sequence lengths, attention becomes the primary time-consuming component. Although quantization has proven to be an effective method for accelerating model inference, existing quantization methods primarily focus on optimizing the linear layer. In response, we first analyze the feasibility of quantization in attention detailedly. Following that, we propose SageAttention, a highly efficient and accurate quantization method for attention. The OPS (operations per second) of our approach outperforms FlashAttention2 and xformers by about 2.1 times and 2.7 times, respectively. SageAttention also achieves superior accuracy performance over FlashAttention3. Comprehensive experiments confirm that our approach incurs almost no end-to-end metrics loss across diverse models, including those for large language processing, image generation, and video generation. The codes are available at https://github.com/thu-ml/SageAttention.","url_abs":"https://arxiv.org/abs/2410.02367v4","url_pdf":"https://arxiv.org/pdf/2410.02367v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sageattention-accurate-8-bit-attention-for","repo_url":"https://github.com/thu-ml/SageAttention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.02367","atlas_url":"https://app.syntology.ai/?focus=2410.02367","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02367"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/thu-ml/SageAttention","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e629740e34804cac","entry":"sageattn","repo":"thu-ml/SageAttention","repo_kind":"official","path":"sageattention/core.py","file_url":"https://github.com/thu-ml/SageAttention/blob/HEAD/sageattention/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e629740e34804cac"}},{"code_sha256_prefix":"8343610f1f3a1e09","entry":"sageattn_qk_int8_pv_fp16_triton","repo":"thu-ml/SageAttention","repo_kind":"official","path":"sageattention/core.py","file_url":"https://github.com/thu-ml/SageAttention/blob/HEAD/sageattention/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8343610f1f3a1e09"}},{"code_sha256_prefix":"2e6007cdb50ffa71","entry":"sageattn_varlen","repo":"thu-ml/SageAttention","repo_kind":"official","path":"sageattention/core.py","file_url":"https://github.com/thu-ml/SageAttention/blob/HEAD/sageattention/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e6007cdb50ffa71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}