{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-22284","title":"Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models","arxiv_id":"2509.22284","date":"2025-09-26","proceeding":"NeurIPS","authors":["Aleksandar Terzić","Nicolas Menet","Michael Hersche","Thomas Hofmann","Abbas Rahimi"],"abstract":"Modern state-space models (SSMs) often utilize transition matrices which enable efficient computation but pose restrictions on the model's expressivity, as measured in terms of the ability to emulate finite-state automata (FSA). While unstructured transition matrices are optimal in terms of expressivity, they come at a prohibitively high compute and memory cost even for moderate state sizes. We propose a structured sparse parametrization of transition matrices in SSMs that enables FSA state tracking with optimal state size and depth, while keeping the computational cost of the recurrence comparable to that of diagonal SSMs. Our method, PD-SSM, parametrizes the transition matrix as the product of a column one-hot matrix ($P$) and a complex-valued diagonal matrix ($D$). Consequently, the computational cost of parallel scans scales linearly with the state size. Theoretically, the model is BIBO-stable and can emulate any $N$-state FSA with one layer of dimension $N$ and a linear readout of size $N \\times N$, significantly improving on all current structured SSM guarantees. Experimentally, the model significantly outperforms a wide collection of modern SSM variants on various FSA state tracking tasks. On multiclass time-series classification, the performance is comparable to that of neural controlled differential equations, a paradigm explicitly built for time-series analysis. Finally, we integrate PD-SSM into a hybrid Transformer-SSM architecture and demonstrate that the model can effectively track the states of a complex FSA in which transitions are encoded as a set of variable-length English sentences. The code is available at https://github.com/IBM/expressive-sparse-state-space-model","url_abs":"https://arxiv.org/abs/2509.22284","url_pdf":"https://arxiv.org/pdf/2509.22284","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.22284","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.22284"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/IBM/expressive-sparse-state-space-model","reach":null}],"summary":{"ran_fixture":3,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"38eba3acd7709579","entry":"binary_operator_efficient","repo":"IBM/expressive-sparse-state-space-model","repo_kind":"found_in_text","path":"time_series_JAX/models/PDSSM.py","file_url":"https://github.com/IBM/expressive-sparse-state-space-model/blob/HEAD/time_series_JAX/models/PDSSM.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38eba3acd7709579"}},{"code_sha256_prefix":"2c0042ac70307731","entry":"forward_pass","repo":"IBM/expressive-sparse-state-space-model","repo_kind":"found_in_text","path":"time_series_JAX/models/PDSSM.py","file_url":"https://github.com/IBM/expressive-sparse-state-space-model/blob/HEAD/time_series_JAX/models/PDSSM.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c0042ac70307731"}},{"code_sha256_prefix":"7dd7b850fe17277a","entry":"indexing_parallel_scan","repo":"IBM/expressive-sparse-state-space-model","repo_kind":"found_in_text","path":"time_series_JAX/models/PDSSM.py","file_url":"https://github.com/IBM/expressive-sparse-state-space-model/blob/HEAD/time_series_JAX/models/PDSSM.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7dd7b850fe17277a"}},{"code_sha256_prefix":"08747f5c6d8bf452","entry":"PDSSMLayer","repo":"IBM/expressive-sparse-state-space-model","repo_kind":"found_in_text","path":"time_series_JAX/models/PDSSM.py","file_url":"https://github.com/IBM/expressive-sparse-state-space-model/blob/HEAD/time_series_JAX/models/PDSSM.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08747f5c6d8bf452"}},{"code_sha256_prefix":"cb7c3068c0365a0b","entry":"custom_function","repo":"IBM/expressive-sparse-state-space-model","repo_kind":"found_in_text","path":"time_series_JAX/models/PDSSM.py","file_url":"https://github.com/IBM/expressive-sparse-state-space-model/blob/HEAD/time_series_JAX/models/PDSSM.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb7c3068c0365a0b"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_api"},"syntology_extracted_results":null}