{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-grammar-learning-trajectories-of-neural","title":"The Grammar-Learning Trajectories of Neural Language Models","arxiv_id":"2109.06096","date":"2021-09-13","proceeding":"ACL 2022 5","authors":["Leshem Choshen","Guy Hacohen","Daphna Weinshall","Omri Abend"],"abstract":"The learning trajectories of linguistic phenomena in humans provide insight into linguistic representation, beyond what can be gleaned from inspecting the behavior of an adult speaker. To apply a similar approach to analyze neural language models (NLM), it is first necessary to establish that different models are similar enough in the generalizations they make. In this paper, we show that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. These findings suggest that there is some mutual inductive bias that underlies these models' learning of linguistic phenomena. Taking inspiration from psycholinguistics, we argue that studying this inductive bias is an opportunity to study the linguistic representation implicit in NLMs. Leveraging these findings, we compare the relative performance on different phenomena at varying learning stages with simpler reference models. Results suggest that NLMs exhibit consistent \"developmental\" stages. Moreover, we find the learning trajectory to be approximately one-dimensional: given an NLM with a certain overall performance, it is possible to predict what linguistic generalizations it has already acquired. Initial analysis of these stages presents phenomena clusters (notably morphological ones), whose performance progresses in unison, suggesting a potential link between the generalizations behind them.","url_abs":"https://arxiv.org/abs/2109.06096v3","url_pdf":"https://arxiv.org/pdf/2109.06096v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-grammar-learning-trajectories-of-neural","repo_url":"https://github.com/borgr/ordert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.06096","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.06096"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/borgr/ordert","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bec434ab039b3074","entry":"accuracy_from_file","repo":"borgr/ordert","repo_kind":"official","path":"transformers/borgr_code/analyse_results.py","file_url":"https://github.com/borgr/ordert/blob/HEAD/transformers/borgr_code/analyse_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bec434ab039b3074"}},{"code_sha256_prefix":"26fd2d193c001284","entry":"average_correlation","repo":"borgr/ordert","repo_kind":"official","path":"transformers/borgr_code/analyse_results.py","file_url":"https://github.com/borgr/ordert/blob/HEAD/transformers/borgr_code/analyse_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26fd2d193c001284"}},{"code_sha256_prefix":"66d59c294b8e06d5","entry":"learnt_orders","repo":"borgr/ordert","repo_kind":"official","path":"transformers/borgr_code/analyse_results.py","file_url":"https://github.com/borgr/ordert/blob/HEAD/transformers/borgr_code/analyse_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66d59c294b8e06d5"}},{"code_sha256_prefix":"33bf519d8bdc6134","entry":"unigram_sentence_prob","repo":"borgr/ordert","repo_kind":"official","path":"kenlm/blimp_eval.py","file_url":"https://github.com/borgr/ordert/blob/HEAD/kenlm/blimp_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33bf519d8bdc6134"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}