{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-08587","title":"Kaczmarz Linear Attention","arxiv_id":"2605.08587","date":"2026-05-09","proceeding":null,"authors":["Jiaxuan Zou","Ruifeng Ren","Yong Liu"],"abstract":"Long-context language modeling remains central to modern sequence modeling, but the quadratic cost of Transformer attention makes scaling computationally prohibitive. Linear recurrent models address this bottleneck by compressing the context into a fixed-size state, making the rule that forgets, writes, and edits information a central design problem. To address state maintenance, Gated DeltaNet (GDN) combines gated state decay with delta-rule residual writes, using a learnable coefficient to balance forgetting and update magnitude. However, this coefficient is learned empirically rather than derived from the underlying objective, which can lead to suboptimal update magnitudes. We revisit the online-regression objective underlying GDN and, inspired by the Kaczmarz projection method, derive the key-norm-normalized dynamic step size $β_t = η_t / (\\|k_t\\|_2^2 + ε)$ for residual updates. We propose Kaczmarz Linear Attention (KLA), a one-scalar modification of GDN that preserves the state shape, gates, linear recurrence, and chunkwise parallel algorithm. At the 0.4B scale with a 1B-token budget, KLA achieves the lowest validation perplexity among evaluated linear-time baselines, 8.09 versus 8.50 for GDN, and remains stable up to 65K tokens. On controlled tasks, KLA reaches 100% on single-needle-in-a-haystack retrieval, improves 8x multi-query associative recall by 7.03 points over GDN, and delivers 2.1x higher decode throughput at 32K context. These results suggest that the key-norm-normalized Kaczmarz coefficient is a first-order design axis for delta-rule sequence models: it improves accuracy, extrapolation, and decoding efficiency without changing the recurrent state or hardware kernel.","url_abs":"https://arxiv.org/abs/2605.08587","url_pdf":"https://arxiv.org/pdf/2605.08587","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2605.08587","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.08587"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/anonymous-kla-review/kla-anonymous","reach":null}],"summary":{"ran":13,"ran_honours":1},"by_repo_kind":{"found_in_text":{"samples":14,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"6526ceac07fa0065","entry":"aggregate_for_plot","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"mqar_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/mqar_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6526ceac07fa0065"}},{"code_sha256_prefix":"736d87b0159dfa9a","entry":"aggregate_mean","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"efficiency_experiment_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/efficiency_experiment_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"736d87b0159dfa9a"}},{"code_sha256_prefix":"973027b79dad3324","entry":"aggregate_mean","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"long_context_extrapolation_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/long_context_extrapolation_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"973027b79dad3324"}},{"code_sha256_prefix":"497efd18eab7050d","entry":"build_relative_and_slope","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"long_context_extrapolation_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/long_context_extrapolation_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"497efd18eab7050d"}},{"code_sha256_prefix":"a30c7463cb6330d7","entry":"build_rope_cache","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"lit_gpt/model.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/lit_gpt/model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a30c7463cb6330d7"}},{"code_sha256_prefix":"e6aada720d62e128","entry":"create_block","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"lit_gpt/model.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/lit_gpt/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e6aada720d62e128"}},{"code_sha256_prefix":"f65b680b6b69a179","entry":"format_metric","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"ablation_47_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/ablation_47_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f65b680b6b69a179"}},{"code_sha256_prefix":"d5c8b7af8c615abe","entry":"get_figures_dir","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"analysis_plot_style.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/analysis_plot_style.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d5c8b7af8c615abe"}},{"code_sha256_prefix":"b0ec0c10e60ad616","entry":"get_model_style","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"analysis_plot_style.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/analysis_plot_style.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b0ec0c10e60ad616"}},{"code_sha256_prefix":"251f453df1ed5d4c","entry":"load_mqar_results","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"ablation_47_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/ablation_47_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"251f453df1ed5d4c"}},{"code_sha256_prefix":"05c508a3b1ef41f8","entry":"load_pretrain_results","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"ablation_47_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/ablation_47_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"05c508a3b1ef41f8"}},{"code_sha256_prefix":"6ffb85ebe4c3c2ba","entry":"load_samples","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"long_context_task_benchmark.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/long_context_task_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6ffb85ebe4c3c2ba"}},{"code_sha256_prefix":"f35fc776f1c781a6","entry":"load_training_val_curves","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"mqar_analysis.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/mqar_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f35fc776f1c781a6"}},{"code_sha256_prefix":"b6c0fb1683b9e6fc","entry":"normalize_model_name","repo":"anonymous-kla-review/kla-anonymous","repo_kind":"found_in_text","path":"analysis_plot_style.py","file_url":"https://github.com/anonymous-kla-review/kla-anonymous/blob/HEAD/analysis_plot_style.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b6c0fb1683b9e6fc"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}