{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-compositionality-of-neural-networks","title":"Compositionality decomposed: how do neural networks generalise?","arxiv_id":"1908.08351","date":"2019-08-22","proceeding":null,"authors":["Dieuwke Hupkes","Verna Dankers","Mathijs Mul","Elia Bruni"],"abstract":"Despite a multitude of empirical studies, little consensus exists on whether neural networks are able to generalise compositionally, a controversy that, in part, stems from a lack of agreement about what it means for a neural model to be compositional. As a response to this controversy, we present a set of tests that provide a bridge between, on the one hand, the vast amount of linguistic and philosophical theory about compositionality of language and, on the other, the successful neural models of language. We collect different interpretations of compositionality and translate them into five theoretically grounded tests for models that are formulated on a task-independent level. In particular, we provide tests to investigate (i) if models systematically recombine known parts and rules (ii) if models can extend their predictions beyond the length they have seen in the training data (iii) if models' composition operations are local or global (iv) if models' predictions are robust to synonym substitutions and (v) if models favour rules or exceptions during training. To demonstrate the usefulness of this evaluation paradigm, we instantiate these five tests on a highly compositional data set which we dub PCFG SET and apply the resulting tests to three popular sequence-to-sequence models: a recurrent, a convolution-based and a transformer model. We provide an in-depth analysis of the results, which uncover the strengths and weaknesses of these three architectures and point to potential areas of improvement.","url_abs":"https://arxiv.org/abs/1908.08351v2","url_pdf":"https://arxiv.org/pdf/1908.08351v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-compositionality-of-neural-networks","repo_url":"https://github.com/i-machine-think/am-i-compositional","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"pcfg-set","name":"PCFG SET","full_name":"Probabilistic Context Free Grammar String Edit Task"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1908.08351","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1908.08351"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/i-machine-think/am-i-compositional","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e9559d69a1b5367f","entry":"count_functions","repo":"i-machine-think/am-i-compositional","repo_kind":"official","path":"src/evaluate/plot_pcfg_statistics.py","file_url":"https://github.com/i-machine-think/am-i-compositional/blob/HEAD/src/evaluate/plot_pcfg_statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9559d69a1b5367f"}},{"code_sha256_prefix":"78ef9e4d235536d3","entry":"depth","repo":"i-machine-think/am-i-compositional","repo_kind":"official","path":"src/evaluate/plot_pcfg_statistics.py","file_url":"https://github.com/i-machine-think/am-i-compositional/blob/HEAD/src/evaluate/plot_pcfg_statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78ef9e4d235536d3"}},{"code_sha256_prefix":"d7c806a14e763121","entry":"depth","repo":"i-machine-think/am-i-compositional","repo_kind":"official","path":"src/evaluate/plot_productivity_statistics.py","file_url":"https://github.com/i-machine-think/am-i-compositional/blob/HEAD/src/evaluate/plot_productivity_statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7c806a14e763121"}},{"code_sha256_prefix":"1e755f5ef00044c8","entry":"interpret","repo":"i-machine-think/am-i-compositional","repo_kind":"official","path":"src/evaluate/plot_overgeneralisation.py","file_url":"https://github.com/i-machine-think/am-i-compositional/blob/HEAD/src/evaluate/plot_overgeneralisation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e755f5ef00044c8"}},{"code_sha256_prefix":"dc6487c8e0261792","entry":"load_results","repo":"i-machine-think/am-i-compositional","repo_kind":"official","path":"src/evaluate/plot_pcfg_statistics.py","file_url":"https://github.com/i-machine-think/am-i-compositional/blob/HEAD/src/evaluate/plot_pcfg_statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc6487c8e0261792"}},{"code_sha256_prefix":"87e48f490224559a","entry":"load_results","repo":"i-machine-think/am-i-compositional","repo_kind":"official","path":"src/evaluate/plot_productivity_statistics.py","file_url":"https://github.com/i-machine-think/am-i-compositional/blob/HEAD/src/evaluate/plot_productivity_statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87e48f490224559a"}},{"code_sha256_prefix":"d0345ff062d64d50","entry":"p_test","repo":"i-machine-think/am-i-compositional","repo_kind":"official","path":"src/evaluate/significance_testing.py","file_url":"https://github.com/i-machine-think/am-i-compositional/blob/HEAD/src/evaluate/significance_testing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0345ff062d64d50"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}