{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-neural-networks-as-recognizers-of","title":"Training Neural Networks as Recognizers of Formal Languages","arxiv_id":"2411.07107","date":"2024-11-11","proceeding":null,"authors":["Alexandra Butoi","Ghazal Khalighinejad","Anej Svete","Josef Valvoda","Ryan Cotterell","Brian DuSell"],"abstract":"Characterizing the computational power of neural network architectures in terms of formal language theory remains a crucial line of research, as it describes lower and upper bounds on the reasoning capabilities of modern AI. However, when empirically testing these bounds, existing work often leaves a discrepancy between experiments and the formal claims they are meant to support. The problem is that formal language theory pertains specifically to recognizers: machines that receive a string as input and classify whether it belongs to a language. On the other hand, it is common to instead use proxy tasks that are similar in only an informal sense, such as language modeling or sequence-to-sequence transduction. We correct this mismatch by training and evaluating neural networks directly as binary classifiers of strings, using a general method that can be applied to a wide variety of languages. As part of this, we extend an algorithm recently proposed by Sn{\\ae}bjarnarson et al. (2024) to do length-controlled sampling of strings from regular languages, with much better asymptotic time complexity than previous methods. We provide results on a variety of languages across the Chomsky hierarchy for three neural architectures: a simple RNN, an LSTM, and a causally-masked transformer. We find that the RNN and LSTM often outperform the transformer, and that auxiliary training objectives such as language modeling can help, although no single objective uniformly improves performance across languages and architectures. Our contributions will facilitate theoretically sound empirical testing of language recognition claims in future work. We have released our datasets as a benchmark called FLaRe (Formal Language Recognition), along with our code.","url_abs":"https://arxiv.org/abs/2411.07107v1","url_pdf":"https://arxiv.org/pdf/2411.07107v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-neural-networks-as-recognizers-of","repo_url":"https://github.com/rycolab/flare","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"training-neural-networks-as-recognizers-of","repo_url":"https://github.com/rycolab/neural-network-recognizers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.07107","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.07107"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rycolab/flare","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rycolab/neural-network-recognizers","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"37ba7b44c2f80edd","entry":"build_softmax_vocab","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/neural_networks/vocabulary.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/neural_networks/vocabulary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37ba7b44c2f80edd"}},{"code_sha256_prefix":"ab34813074cc49bf","entry":"divide","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/analysis/plot_cross_entropy_vs_num_edits.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/analysis/plot_cross_entropy_vs_num_edits.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab34813074cc49bf"}},{"code_sha256_prefix":"4785d72514a2e8c3","entry":"generate_batches","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/neural_networks/training_loop.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/neural_networks/training_loop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4785d72514a2e8c3"}},{"code_sha256_prefix":"ab6fb5ddcfea8962","entry":"get_file_names_from_directory","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/neural_networks/prepare_data.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/neural_networks/prepare_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab6fb5ddcfea8962"}},{"code_sha256_prefix":"dfd7c6df71691537","entry":"get_loss_terms","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/neural_networks/training_loop.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/neural_networks/training_loop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfd7c6df71691537"}},{"code_sha256_prefix":"357c7b74e93367d1","entry":"get_smoothed_points","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/analysis/plot_cross_entropy_vs_length.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/analysis/plot_cross_entropy_vs_length.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"357c7b74e93367d1"}},{"code_sha256_prefix":"8aeac5c10121fbe0","entry":"get_test_accuracy","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/analysis/plot_cross_entropy_vs_length.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/analysis/plot_cross_entropy_vs_length.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aeac5c10121fbe0"}},{"code_sha256_prefix":"08bd77bfa41e19db","entry":"load_prepared_labels_file","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/neural_networks/data.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/neural_networks/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08bd77bfa41e19db"}},{"code_sha256_prefix":"c83517b2f4da4069","entry":"load_vocabulary_data_from_file","repo":"rycolab/neural-network-recognizers","repo_kind":"official","path":"src/recognizers/neural_networks/vocabulary.py","file_url":"https://github.com/rycolab/neural-network-recognizers/blob/HEAD/src/recognizers/neural_networks/vocabulary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c83517b2f4da4069"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}