{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/implicit-language-models-are-rnns-balancing","title":"Implicit Language Models are RNNs: Balancing Parallelization and Expressivity","arxiv_id":"2502.07827","date":"2025-02-10","proceeding":null,"authors":["Mark Schöne","Babak Rahmani","Heiner Kremer","Fabian Falck","Hitesh Ballani","Jannes Gladrow"],"abstract":"State-space models (SSMs) and transformers dominate the language modeling landscape. However, they are constrained to a lower computational complexity than classical recurrent neural networks (RNNs), limiting their expressivity. In contrast, RNNs lack parallelization during training, raising fundamental questions about the trade off between parallelization and expressivity. We propose implicit SSMs, which iterate a transformation until convergence to a fixed point. Theoretically, we show that implicit SSMs implement the non-linear state-transitions of RNNs. Empirically, we find that only approximate fixed-point convergence suffices, enabling the design of a scalable training curriculum that largely retains parallelization, with full convergence required only for a small subset of tokens. Our approach demonstrates superior state-tracking capabilities on regular languages, surpassing transformers and SSMs. We further scale implicit SSMs to natural language reasoning tasks and pretraining of large-scale language models up to 1.3B parameters on 207B tokens representing, to our knowledge, the largest implicit model trained to date. Notably, our implicit models outperform their explicit counterparts on standard benchmarks. Our code is publicly available at http://github.com/microsoft/implicit_languagemodels .","url_abs":"https://arxiv.org/abs/2502.07827v3","url_pdf":"https://arxiv.org/pdf/2502.07827v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"implicit-language-models-are-rnns-balancing","repo_url":"https://github.com/microsoft/implicit_languagemodels","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.07827","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.07827"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/implicit_languagemodels","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"97496330a30f8c79","entry":"apply_mask_to_padding_states","repo":"microsoft/implicit_languagemodels","repo_kind":"official","path":"implicit_llm/modules/mamba2.py","file_url":"https://github.com/microsoft/implicit_languagemodels/blob/HEAD/implicit_llm/modules/mamba2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97496330a30f8c79"}},{"code_sha256_prefix":"eb10ae6a5183f56d","entry":"get_mlp_cls","repo":"microsoft/implicit_languagemodels","repo_kind":"official","path":"implicit_llm/modules/mlp.py","file_url":"https://github.com/microsoft/implicit_languagemodels/blob/HEAD/implicit_llm/modules/mlp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb10ae6a5183f56d"}},{"code_sha256_prefix":"541ba8da0f7b6512","entry":"repeat_kv","repo":"microsoft/implicit_languagemodels","repo_kind":"official","path":"implicit_llm/modules/attention.py","file_url":"https://github.com/microsoft/implicit_languagemodels/blob/HEAD/implicit_llm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"541ba8da0f7b6512"}},{"code_sha256_prefix":"35f62568f91aaa71","entry":"sdpa_attention_forward","repo":"microsoft/implicit_languagemodels","repo_kind":"official","path":"implicit_llm/modules/attention.py","file_url":"https://github.com/microsoft/implicit_languagemodels/blob/HEAD/implicit_llm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35f62568f91aaa71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}