{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/native-sparse-attention-hardware-aligned-and","title":"Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention","arxiv_id":"2502.11089","date":"2025-02-16","proceeding":null,"authors":["Jingyang Yuan","Huazuo Gao","Damai Dai","Junyu Luo","Liang Zhao","Zhengyan Zhang","Zhenda Xie","Y. X. Wei","Lean Wang","Zhiping Xiao","Yuqing Wang","Chong Ruan","Ming Zhang","Wenfeng Liang","Wangding Zeng"],"abstract":"Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations to achieve efficient long-context modeling. NSA employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision. Our approach advances sparse attention design with two key innovations: (1) We achieve substantial speedups through arithmetic intensity-balanced algorithm design, with implementation optimizations for modern hardware. (2) We enable end-to-end training, reducing pretraining computation without sacrificing model performance. As shown in Figure 1, experiments show the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves substantial speedups over Full Attention on 64k-length sequences across decoding, forward propagation, and backward propagation, validating its efficiency throughout the model lifecycle.","url_abs":"https://arxiv.org/abs/2502.11089v1","url_pdf":"https://arxiv.org/pdf/2502.11089v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"native-sparse-attention-hardware-aligned-and","repo_url":"https://github.com/britacatalin/bsa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"native-sparse-attention-hardware-aligned-and","repo_url":"https://github.com/microsoft/seerattention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"native-sparse-attention-hardware-aligned-and","repo_url":"https://github.com/sandai-org/magiattention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.11089","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.11089"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/britacatalin/bsa","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/seerattention","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sandai-org/magiattention","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":15},"by_repo_kind":{"listed":{"samples":19,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b93448058eb3a756","entry":"NativeSparseAttention","repo":"britacatalin/bsa","repo_kind":"listed","path":"nsa/native_sparse_attention.py","file_url":"https://github.com/britacatalin/bsa/blob/HEAD/nsa/native_sparse_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b93448058eb3a756"}},{"code_sha256_prefix":"f36889830d661d6f","entry":"StaticGatedAttention","repo":"britacatalin/bsa","repo_kind":"listed","path":"nsa/native_sparse_attention.py","file_url":"https://github.com/britacatalin/bsa/blob/HEAD/nsa/native_sparse_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f36889830d661d6f"}},{"code_sha256_prefix":"d9f141ee5201879b","entry":"TokenGatedAttention","repo":"britacatalin/bsa","repo_kind":"listed","path":"nsa/native_sparse_attention.py","file_url":"https://github.com/britacatalin/bsa/blob/HEAD/nsa/native_sparse_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9f141ee5201879b"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"microsoft/seerattention","repo_kind":"listed","path":"seer_attn/modules/common.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/seer_attn/modules/common.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"202aa55ac60ff1f6","entry":"apply_chat_template","repo":"microsoft/seerattention","repo_kind":"listed","path":"distillation_decode.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/distillation_decode.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"202aa55ac60ff1f6"}},{"code_sha256_prefix":"8c7b215a2004bfc6","entry":"apply_rotary_pos_emb","repo":"microsoft/seerattention","repo_kind":"listed","path":"seer_attn/modules/common.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/seer_attn/modules/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c7b215a2004bfc6"}},{"code_sha256_prefix":"89cf2076c1b8ce86","entry":"apply_rotary_pos_emb_single","repo":"microsoft/seerattention","repo_kind":"listed","path":"seer_attn/modules/common.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/seer_attn/modules/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89cf2076c1b8ce86"}},{"code_sha256_prefix":"a8c8717e85efa3ea","entry":"average_groups","repo":"microsoft/seerattention","repo_kind":"listed","path":"eval/longbench/average_sparsity.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/eval/longbench/average_sparsity.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8c8717e85efa3ea"}},{"code_sha256_prefix":"4334f4eeb6104bee","entry":"build_optimized_chunks","repo":"microsoft/seerattention","repo_kind":"listed","path":"distillation_decode.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/distillation_decode.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4334f4eeb6104bee"}},{"code_sha256_prefix":"e9c41059655ed7e3","entry":"calculate_averages_by_range","repo":"microsoft/seerattention","repo_kind":"listed","path":"eval/longbench/average_score.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/eval/longbench/average_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9c41059655ed7e3"}},{"code_sha256_prefix":"8498177d23b25d59","entry":"generate_base_sparsity_mask","repo":"microsoft/seerattention","repo_kind":"listed","path":"eval/efficiency/benchmark_sparse_attn.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/eval/efficiency/benchmark_sparse_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8498177d23b25d59"}},{"code_sha256_prefix":"09bb2cb546d9e69e","entry":"get_tensors","repo":"microsoft/seerattention","repo_kind":"listed","path":"eval/efficiency/benchmark_sparse_attn.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/eval/efficiency/benchmark_sparse_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09bb2cb546d9e69e"}},{"code_sha256_prefix":"2ce8fcf5bedd19ec","entry":"layer_norm_ref","repo":"microsoft/seerattention","repo_kind":"listed","path":"seer_attn/modules/layernorm.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/seer_attn/modules/layernorm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ce8fcf5bedd19ec"}},{"code_sha256_prefix":"cc892598a14c82a5","entry":"load_json","repo":"microsoft/seerattention","repo_kind":"listed","path":"eval/longbench/average_score.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/eval/longbench/average_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc892598a14c82a5"}},{"code_sha256_prefix":"ea5becf731d3b58d","entry":"ref_program_fa","repo":"microsoft/seerattention","repo_kind":"listed","path":"eval/efficiency/decode_kernel_eval.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/eval/efficiency/decode_kernel_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea5becf731d3b58d"}},{"code_sha256_prefix":"ab802ae3195c6316","entry":"rms_norm_ref","repo":"microsoft/seerattention","repo_kind":"listed","path":"seer_attn/modules/layernorm.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/seer_attn/modules/layernorm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab802ae3195c6316"}},{"code_sha256_prefix":"046184b390b18314","entry":"run_benchmark","repo":"microsoft/seerattention","repo_kind":"listed","path":"eval/efficiency/benchmark_sparse_attn.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/eval/efficiency/benchmark_sparse_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"046184b390b18314"}},{"code_sha256_prefix":"3ef366d8b91a362a","entry":"tokenize_fn","repo":"microsoft/seerattention","repo_kind":"listed","path":"distillation_decode.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/distillation_decode.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ef366d8b91a362a"}},{"code_sha256_prefix":"f9963e3bc9860eda","entry":"tokenize_fn","repo":"microsoft/seerattention","repo_kind":"listed","path":"distillation_prefill.py","file_url":"https://github.com/microsoft/seerattention/blob/HEAD/distillation_prefill.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9963e3bc9860eda"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}