{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lower-bounds-on-the-expressivity-of-recurrent","title":"Lower Bounds on the Expressivity of Recurrent Neural Language Models","arxiv_id":"2405.19222","date":"2024-05-29","proceeding":null,"authors":["Anej Svete","Franz Nowak","Anisha Mohamed Sahabdeen","Ryan Cotterell"],"abstract":"The recent successes and spread of large neural language models (LMs) call for a thorough understanding of their computational ability. Describing their computational abilities through LMs' \\emph{representational capacity} is a lively area of research. However, investigation into the representational capacity of neural LMs has predominantly focused on their ability to \\emph{recognize} formal languages. For example, recurrent neural networks (RNNs) with Heaviside activations are tightly linked to regular languages, i.e., languages defined by finite-state automata (FSAs). Such results, however, fall short of describing the capabilities of RNN \\emph{language models} (LMs), which are definitionally \\emph{distributions} over strings. We take a fresh look at the representational capacity of RNN LMs by connecting them to \\emph{probabilistic} FSAs and demonstrate that RNN LMs with linearly bounded precision can express arbitrary regular LMs.","url_abs":"https://arxiv.org/abs/2405.19222v2","url_pdf":"https://arxiv.org/pdf/2405.19222v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lower-bounds-on-the-expressivity-of-recurrent","repo_url":"https://github.com/rycolab/nondeterministic-rnns","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.19222","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}