{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-illusion-of-state-in-state-space-models","title":"The Illusion of State in State-Space Models","arxiv_id":"2404.08819","date":"2024-04-12","proceeding":null,"authors":["William Merrill","Jackson Petty","Ashish Sabharwal"],"abstract":"State-space models (SSMs) have emerged as a potential alternative architecture for building large language models (LLMs) compared to the previously ubiquitous transformer architecture. One theoretical weakness of transformers is that they cannot express certain kinds of sequential computation and state tracking (Merrill & Sabharwal, 2023), which SSMs are explicitly designed to address via their close architectural similarity to recurrent neural networks (RNNs). But do SSMs truly have an advantage (over transformers) in expressive power for state tracking? Surprisingly, the answer is no. Our analysis reveals that the expressive power of SSMs is limited very similarly to transformers: SSMs cannot express computation outside the complexity class $\\mathsf{TC}^0$. In particular, this means they cannot solve simple state-tracking problems like permutation composition. It follows that SSMs are provably unable to accurately track chess moves with certain notation, evaluate code, or track entities in a long narrative. To supplement our formal analysis, we report experiments showing that Mamba-style SSMs indeed struggle with state tracking. Thus, despite its recurrent formulation, the \"state\" in an SSM is an illusion: SSMs have similar expressiveness limitations to non-recurrent models like transformers, which may fundamentally limit their ability to solve real-world state-tracking problems.","url_abs":"https://arxiv.org/abs/2404.08819v3","url_pdf":"https://arxiv.org/pdf/2404.08819v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-illusion-of-state-in-state-space-models","repo_url":"https://github.com/benjamin-walker/selective-ssms-and-linear-cdes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.08819","atlas_url":"https://app.syntology.ai/?focus=2404.08819","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.08819"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/benjamin-walker/selective-ssms-and-linear-cdes","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran_violates":1},"by_repo_kind":{"listed":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"6d8a954518022c0b","entry":"load_config","repo":"benjamin-walker/selective-ssms-and-linear-cdes","repo_kind":"listed","path":"run_experiment.py","file_url":"https://github.com/benjamin-walker/selective-ssms-and-linear-cdes/blob/HEAD/run_experiment.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d8a954518022c0b"}},{"code_sha256_prefix":"56e94810331e4206","entry":"obtain_features_from_model","repo":"benjamin-walker/selective-ssms-and-linear-cdes","repo_kind":"listed","path":"linear_cde.py","file_url":"https://github.com/benjamin-walker/selective-ssms-and-linear-cdes/blob/HEAD/linear_cde.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56e94810331e4206"}},{"code_sha256_prefix":"0209e6a9094c9a29","entry":"scan_cde_solve","repo":"benjamin-walker/selective-ssms-and-linear-cdes","repo_kind":"listed","path":"linear_cde.py","file_url":"https://github.com/benjamin-walker/selective-ssms-and-linear-cdes/blob/HEAD/linear_cde.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0209e6a9094c9a29"}},{"code_sha256_prefix":"92c5aec2331fa7e1","entry":"str2bool","repo":"benjamin-walker/selective-ssms-and-linear-cdes","repo_kind":"listed","path":"run_experiment.py","file_url":"https://github.com/benjamin-walker/selective-ssms-and-linear-cdes/blob/HEAD/run_experiment.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92c5aec2331fa7e1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}