{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deltaproduct-increasing-the-expressivity-of","title":"DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products","arxiv_id":"2502.10297","date":"2025-02-14","proceeding":null,"authors":["Julien Siems","Timur Carstensen","Arber Zela","Frank Hutter","Massimiliano Pontil","Riccardo Grazzi"],"abstract":"Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference. However, existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices. Diagonal matrices, used in models such as Mamba, GLA, or mLSTM, yield fast runtime but have limited expressivity. To address this, recent architectures such as DeltaNet and RWKV-7 adopted a diagonal plus rank-1 structure, which allows simultaneous token and channel mixing, improving associative recall and, as recently shown, state-tracking when allowing negative eigenvalues in the state-transition matrices. Building on the interpretation of DeltaNet's recurrence as performing one step of online gradient descent per token on an associative recall loss, we introduce DeltaProduct, which instead takes multiple ($n_h$) steps per token. This naturally leads to diagonal plus rank-$n_h$ state-transition matrices, formed as products of $n_h$ generalized Householder transformations, providing a tunable mechanism to balance expressivity and efficiency. We provide a detailed theoretical characterization of the state-tracking capability of DeltaProduct in finite precision, showing how it improves by increasing $n_h$. Our extensive experiments demonstrate that DeltaProduct outperforms DeltaNet in both state-tracking and language modeling, while also showing significantly improved length extrapolation capabilities.","url_abs":"https://arxiv.org/abs/2502.10297v6","url_pdf":"https://arxiv.org/pdf/2502.10297v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deltaproduct-increasing-the-expressivity-of","repo_url":"https://github.com/sustcsonglin/flash-linear-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mamba","task_name":"Mamba"}],"methods":[{"method_slug":"mamba","method_name":"Mamba"},{"method_slug":"mlstm","method_name":"mLSTM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.10297","atlas_url":"https://app.syntology.ai/?focus=2502.10297","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.10297"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sustcsonglin/flash-linear-attention","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"community":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ee8f82afd691729b","entry":"elu_p1","repo":"automl/deltaproduct","repo_kind":"community","path":"flash-linear-attention/fla/layers/gated_deltaproduct.py","file_url":"https://github.com/automl/deltaproduct/blob/HEAD/flash-linear-attention/fla/layers/gated_deltaproduct.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee8f82afd691729b"}},{"code_sha256_prefix":"e544fdb6373b9f42","entry":"sum_norm","repo":"automl/deltaproduct","repo_kind":"community","path":"flash-linear-attention/fla/layers/gated_deltaproduct.py","file_url":"https://github.com/automl/deltaproduct/blob/HEAD/flash-linear-attention/fla/layers/gated_deltaproduct.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e544fdb6373b9f42"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}