{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-21325","title":"Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers","arxiv_id":"2605.21325","date":"2026-05-20","proceeding":null,"authors":["Aleksandros Sobczyk","Gioele Gottardo","Christos K. Matzoros","Mirko De Vita","Filip Skogh","Anastasios Zouzias","Jiawei Zhuang"],"abstract":"Linear attention has emerged as a cornerstone for efficient long-context architectures, as evidenced by its integration into state-of-the-art open-source models including Qwen3.5/3.6, Kimi Linear, and RWKV-7. Models that incorporate linear attention layers with the so-called Delta-Rule involve the inversion of triangular matrices as a core sub-routine. This operation often forms a performance bottleneck, and, due to its high-sensitivity to numerical errors, it can significantly deteriorate end-to-end model accuracy if it is not carefully implemented. This work provides a systematic analysis of both direct and iterative triangular inversion algorithms, targeting methods that are rich in matrix products, and, therefore, have the potential to efficiently utilize modern hardware. To that end, our analysis covers a broad spectrum of mathematical and practical aspects, with a heavy focus on numerical stability, computational complexity, and, ultimately, hardware efficiency and practical considerations. We provide a rigorous experimental evaluation to verify these properties in practical scenarios, and in low-precision floating-point representations, highlighting the strengths and limitations of each method. Performance benchmarks on NPUs reveal up to $4.3\\times$ speed-up against the state-of-the-art implementations of SGLang for triangular matrix inversion, leading to significant performance improvements on the entire layer level, while maintaining full end-to-end model accuracy.","url_abs":"https://arxiv.org/abs/2605.21325","url_pdf":"https://arxiv.org/pdf/2605.21325","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.21325","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.21325"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/huawei-csl/pto-kernels","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/huawei-csl/gdn-tri-inverse","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/huawei-csl/megagdn-pto","reach":null}],"summary":{"ran":15,"unverified":13},"by_repo_kind":{"found_in_text":{"samples":28,"ran":15,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":28,"samples":[{"code_sha256_prefix":"3be586d513227377","entry":"bind_triton","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"vllm_patch/chunk_kda_pto.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/vllm_patch/chunk_kda_pto.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3be586d513227377"}},{"code_sha256_prefix":"8c58a327ad934418","entry":"chunk_gated_delta_rule_native","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/triton_chunk_gdn/fla/chunk.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/triton_chunk_gdn/fla/chunk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"8c58a327ad934418"}},{"code_sha256_prefix":"88c21f1f458ccf5c","entry":"chunk_kda_pto","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"vllm_patch/chunk_kda_pto.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/vllm_patch/chunk_kda_pto.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88c21f1f458ccf5c"}},{"code_sha256_prefix":"49cd9155190e49b2","entry":"compile_kernel","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/pto_demo_utils.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/pto_demo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"49cd9155190e49b2"}},{"code_sha256_prefix":"179b4ecb6419827a","entry":"get_tp_context","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"docker/overrides/model_runner_v1.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/docker/overrides/model_runner_v1.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"179b4ecb6419827a"}},{"code_sha256_prefix":"65c98279ed930932","entry":"inv_tril_inplace","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/triton_chunk_gdn/fla/chunk.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/triton_chunk_gdn/fla/chunk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"65c98279ed930932"}},{"code_sha256_prefix":"29854c7c42411c2b","entry":"npu_from_cpu","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/dynamic_multi_core/a5/run_kernel_ctypes.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/dynamic_multi_core/a5/run_kernel_ctypes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"29854c7c42411c2b"}},{"code_sha256_prefix":"43739c7f7a9f008f","entry":"npu_tensor","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/dynamic_multi_core/a5/run_kernel_pybind.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/dynamic_multi_core/a5/run_kernel_pybind.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"43739c7f7a9f008f"}},{"code_sha256_prefix":"7140e33bf7bef21f","entry":"summarize","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/benchmark.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"7140e33bf7bef21f"}},{"code_sha256_prefix":"6c19edb11fb6754c","entry":"tensor_ptr","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/pto_demo_utils.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/pto_demo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"6c19edb11fb6754c"}},{"code_sha256_prefix":"e4e5891b8b4488b1","entry":"total_chunks","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"megagdn_pto/kernel_libs.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/megagdn_pto/kernel_libs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4e5891b8b4488b1"}},{"code_sha256_prefix":"33a4536ecf79ed25","entry":"transpose_beta","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"megagdn_pto/kernel_libs.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/megagdn_pto/kernel_libs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33a4536ecf79ed25"}},{"code_sha256_prefix":"e0779707b00e56f4","entry":"transpose_gates","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"megagdn_pto/kernel_libs.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/megagdn_pto/kernel_libs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0779707b00e56f4"}},{"code_sha256_prefix":"27aa399aa0968a73","entry":"tri_inv_qwen3_next_default","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/linalg.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/linalg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"27aa399aa0968a73"}},{"code_sha256_prefix":"bf1d857bd30bc516","entry":"workspace","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/run_mix_ctypes.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/run_mix_ctypes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"bf1d857bd30bc516"}},{"code_sha256_prefix":"b26eb7a5b59d8e1f","entry":"add_reference_to_path","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/pto_demo_utils.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/pto_demo_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"b26eb7a5b59d8e1f"}},{"code_sha256_prefix":"bcc14bcd0eec3750","entry":"build_kernel","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/run_kernel_pybind.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/run_kernel_pybind.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"bcc14bcd0eec3750"}},{"code_sha256_prefix":"1a65bd82810630c4","entry":"compile_chunk_kernel","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"megagdn_pto/compile.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/megagdn_pto/compile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a65bd82810630c4"}},{"code_sha256_prefix":"8991055d8785b3a2","entry":"compile_tri_inverse","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"megagdn_pto/compile.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/megagdn_pto/compile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8991055d8785b3a2"}},{"code_sha256_prefix":"b0bfd6df1362ea99","entry":"event_bench","repo":"huawei-csl/pto-kernels","repo_kind":"found_in_text","path":".skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/benchmark.py","file_url":"https://github.com/huawei-csl/pto-kernels/blob/HEAD/.skills/testing-pto-kernels/reference/dynamic_multi_core/a2a3/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"b0bfd6df1362ea99"}},{"code_sha256_prefix":"3872a6d7e9a842a6","entry":"fast_inv_tril","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/triton_chunk_gdn/fla/chunk.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/triton_chunk_gdn/fla/chunk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"3872a6d7e9a842a6"}},{"code_sha256_prefix":"94a3d31751316112","entry":"recurrent_gated_delta_rule_ref","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/core.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"94a3d31751316112"}},{"code_sha256_prefix":"1902ecb472db2fcc","entry":"ref_chunk_cumsum","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_chunk_cumsum.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_chunk_cumsum.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"1902ecb472db2fcc"}},{"code_sha256_prefix":"2996cf978e93d4a1","entry":"ref_chunk_h","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_chunk_h.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_chunk_h.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"2996cf978e93d4a1"}},{"code_sha256_prefix":"7136a894cedf2d88","entry":"ref_chunk_o","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_chunk_o.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_chunk_o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"7136a894cedf2d88"}},{"code_sha256_prefix":"f3d36efe796d3c18","entry":"ref_seq_gdn","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_full.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_full.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"f3d36efe796d3c18"}},{"code_sha256_prefix":"18bceabcfb10e5ac","entry":"ref_wy_fast","repo":"huawei-csl/gdn-tri-inverse","repo_kind":"found_in_text","path":"src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_wy_fast.py","file_url":"https://github.com/huawei-csl/gdn-tri-inverse/blob/HEAD/src/gdn_tri_inverse/tilelang_chunk_gdn/opt_gdn_wy_fast.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"18bceabcfb10e5ac"}},{"code_sha256_prefix":"04b68677ccfb33a0","entry":"staged_separate_launch","repo":"huawei-csl/megagdn-pto","repo_kind":"found_in_text","path":"benchmarks/kernel/bench_pto_pipeline_latency.py","file_url":"https://github.com/huawei-csl/megagdn-pto/blob/HEAD/benchmarks/kernel/bench_pto_pipeline_latency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04b68677ccfb33a0"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}