{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sageattention2-technical-report-accurate-4","title":"SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization","arxiv_id":"2411.10958","date":"2024-11-17","proceeding":null,"authors":["Jintao Zhang","Haofeng Huang","Pengle Zhang","Jia Wei","Jun Zhu","Jianfei Chen"],"abstract":"Although quantization for linear layers has been widely used, its application to accelerate the attention process remains limited. To further enhance the efficiency of attention computation compared to SageAttention while maintaining precision, we propose SageAttention2, which utilizes significantly faster 4-bit matrix multiplication (Matmul) alongside additional precision-enhancing techniques. First, we propose to quantize matrices $(Q, K)$ to INT4 in a hardware-friendly thread-level granularity and quantize matrices $(\\widetilde P, V)$ to FP8. Second, we propose a method to smooth $Q$, enhancing the accuracy of INT4 $QK^\\top$. Third, we propose a two-level accumulation strategy for $\\widetilde PV$ to enhance the accuracy of FP8 $\\widetilde PV$. The operations per second (OPS) of SageAttention2 surpass FlashAttention2 and xformers by about 3x and 4.5x on RTX4090, respectively. Moreover, SageAttention2 matches the speed of FlashAttention3(fp8) on the Hopper GPUs, while delivering much higher accuracy. Comprehensive experiments confirm that our approach incurs negligible end-to-end metrics loss across diverse models, including those for language, image, and video generation. The code is available at https://github.com/thu-ml/SageAttention.","url_abs":"https://arxiv.org/abs/2411.10958v6","url_pdf":"https://arxiv.org/pdf/2411.10958v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sageattention2-technical-report-accurate-4","repo_url":"https://github.com/thu-ml/SageAttention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"sageattention2-technical-report-accurate-4","repo_url":"https://github.com/thu-ml/spargeattn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.10958","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.10958"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-ml/SageAttention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-ml/spargeattn","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"e629740e34804cac","entry":"sageattn","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e629740e34804cac"}},{"code_sha256_prefix":"0d53f0eb082706d1","entry":"sageattn","repo":"thu-ml/sageattention","repo_kind":"official","path":"sageattention/core.py","file_url":"https://github.com/thu-ml/sageattention/blob/HEAD/sageattention/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d53f0eb082706d1"}},{"code_sha256_prefix":"8343610f1f3a1e09","entry":"sageattn_qk_int8_pv_fp16_triton","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8343610f1f3a1e09"}},{"code_sha256_prefix":"b342df7dc419a801","entry":"sageattn_qk_int8_pv_fp16_triton","repo":"thu-ml/sageattention","repo_kind":"official","path":"sageattention/core.py","file_url":"https://github.com/thu-ml/sageattention/blob/HEAD/sageattention/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b342df7dc419a801"}},{"code_sha256_prefix":"2e6007cdb50ffa71","entry":"sageattn_varlen","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"2e6007cdb50ffa71"}},{"code_sha256_prefix":"c9ff307a043756a1","entry":"sageattn_varlen","repo":"thu-ml/sageattention","repo_kind":"official","path":"sageattention/core.py","file_url":"https://github.com/thu-ml/sageattention/blob/HEAD/sageattention/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9ff307a043756a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}