{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-makes-a-language-easy-to-deep-learn","title":"What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure","arxiv_id":"2302.12239","date":"2023-02-23","proceeding":null,"authors":["Lukas Galke","Yoav Ram","Limor Raviv"],"abstract":"Deep neural networks drive the success of natural language processing. A fundamental property of language is its compositional structure, allowing humans to systematically produce forms for new meanings. For humans, languages with more compositional and transparent structures are typically easier to learn than those with opaque and irregular structures. However, this learnability advantage has not yet been shown for deep neural networks, limiting their use as models for human language learning. Here, we directly test how neural networks compare to humans in learning and generalizing different languages that vary in their degree of compositional structure. We evaluate the memorization and generalization capabilities of a large language model and recurrent neural networks, and show that both deep neural networks exhibit a learnability advantage for more structured linguistic input: neural networks exposed to more compositional languages show more systematic generalization, greater agreement between different agents, and greater similarity to human learners.","url_abs":"https://arxiv.org/abs/2302.12239v4","url_pdf":"https://arxiv.org/pdf/2302.12239v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-makes-a-language-easy-to-deep-learn","repo_url":"https://github.com/lgalke/easy2deeplearn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"systematic-generalization","task_name":"Systematic Generalization"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"test","method_name":"Test"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.12239","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.12239"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lgalke/easy2deeplearn","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":4},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2f878799bcb9b1aa","entry":"add_structure_bin","repo":"lgalke/easy2deeplearn","repo_kind":"official","path":"stats.py","file_url":"https://github.com/lgalke/easy2deeplearn/blob/HEAD/stats.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f878799bcb9b1aa"}},{"code_sha256_prefix":"11aa39fab26c3a48","entry":"add_structure_score","repo":"lgalke/easy2deeplearn","repo_kind":"official","path":"stats.py","file_url":"https://github.com/lgalke/easy2deeplearn/blob/HEAD/stats.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11aa39fab26c3a48"}},{"code_sha256_prefix":"272b2ba41cdd272a","entry":"make_example","repo":"lgalke/easy2deeplearn","repo_kind":"official","path":"lang2prompt.py","file_url":"https://github.com/lgalke/easy2deeplearn/blob/HEAD/lang2prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"272b2ba41cdd272a"}},{"code_sha256_prefix":"246b6d3de7325350","entry":"make_prompt","repo":"lgalke/easy2deeplearn","repo_kind":"official","path":"lang2prompt.py","file_url":"https://github.com/lgalke/easy2deeplearn/blob/HEAD/lang2prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"246b6d3de7325350"}},{"code_sha256_prefix":"f486a194a7e381e6","entry":"prepare_example_for_completion","repo":"lgalke/easy2deeplearn","repo_kind":"official","path":"lang2prompt.py","file_url":"https://github.com/lgalke/easy2deeplearn/blob/HEAD/lang2prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f486a194a7e381e6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}