{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memorize-or-generalize-searching-for-a","title":"Memorize or generalize? Searching for a compositional RNN in a haystack","arxiv_id":"1802.06467","date":"2018-02-18","proceeding":null,"authors":["Adam Liška","Germán Kruszewski","Marco Baroni"],"abstract":"Neural networks are very powerful learning systems, but they do not readily\ngeneralize from one task to the other. This is partly due to the fact that they\ndo not learn in a compositional way, that is, by discovering skills that are\nshared by different tasks, and recombining them to solve new problems. In this\npaper, we explore the compositional generalization capabilities of recurrent\nneural networks (RNNs). We first propose the lookup table composition domain as\na simple setup to test compositional behaviour and show that it is\ntheoretically possible for a standard RNN to learn to behave compositionally in\nthis domain when trained with standard gradient descent and provided with\nadditional supervision. We then remove this additional supervision and perform\na search over a large number of model initializations to investigate the\nproportion of RNNs that can still converge to a compositional solution. We\ndiscover that a small but non-negligible proportion of RNNs do reach partial\ncompositional solutions even without special architectural constraints. This\nsuggests that a combination of gradient descent and evolutionary strategies\ndirectly favouring the minority models that developed more compositional\napproaches might suffice to lead standard RNNs towards compositional solutions.","url_abs":"http://arxiv.org/abs/1802.06467v2","url_pdf":"http://arxiv.org/pdf/1802.06467v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memorize-or-generalize-searching-for-a","repo_url":"https://github.com/i-machine-think/machine-tasks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1802.06467","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}