{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-does-compositional-structure-yield","title":"When does compositional structure yield compositional generalization? A kernel theory","arxiv_id":"2405.16391","date":"2024-05-26","proceeding":null,"authors":["Samuel Lippl","Kim Stachenfeld"],"abstract":"Compositional generalization (the ability to respond correctly to novel combinations of familiar components) is thought to be a cornerstone of intelligent behavior. Compositionally structured (e.g. disentangled) representations support this ability; however, the conditions under which they are sufficient for the emergence of compositional generalization remain unclear. To address this gap, we present a theory of compositional generalization in kernel models with fixed, compositionally structured representations. This provides a tractable framework for characterizing the impact of training data statistics on generalization. We find that these models are limited to functions that assign values to each combination of components seen during training, and then sum up these values (\"conjunction-wise additivity\"). This imposes fundamental restrictions on the set of tasks compositionally structured kernel models can learn, in particular preventing them from transitively generalizing equivalence relations. Even for compositional tasks that they can learn in principle, we identify novel failure modes in compositional generalization (memorization leak and shortcut bias) that arise from biases in the training data. Finally, we empirically validate our theory, showing that it captures the behavior of deep neural networks (convolutional networks, residual networks, and Vision Transformers) trained on a set of compositional tasks with similarly structured data. Ultimately, this work examines how statistical structure in the training data can affect compositional generalization, with implications for how to identify and remedy failure modes in deep learning models.","url_abs":"https://arxiv.org/abs/2405.16391v3","url_pdf":"https://arxiv.org/pdf/2405.16391v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-does-compositional-structure-yield","repo_url":"https://github.com/sflippl/compositional-generalization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.16391","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16391"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sflippl/compositional-generalization","reach":null}],"summary":{"ran":6,"ran_honours":1,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"8d0ab110639937d8","entry":"CSVDataFrame","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d0ab110639937d8"}},{"code_sha256_prefix":"a8f89e3acd77db3a","entry":"binary_accuracy","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8f89e3acd77db3a"}},{"code_sha256_prefix":"9ae529d03358d2a1","entry":"complete_sims","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ae529d03358d2a1"}},{"code_sha256_prefix":"329d4d83983cce32","entry":"is_pos_def","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"329d4d83983cce32"}},{"code_sha256_prefix":"4dd4ca690ab53814","entry":"kernel","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4dd4ca690ab53814"}},{"code_sha256_prefix":"5c2f9837b73be1db","entry":"margin","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c2f9837b73be1db"}},{"code_sha256_prefix":"577f80aee94bb229","entry":"mse","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"577f80aee94bb229"}},{"code_sha256_prefix":"83c8b732856a57d7","entry":"sims_pos_def","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83c8b732856a57d7"}},{"code_sha256_prefix":"acb11a7bf4474e2f","entry":"KernelMachine","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"acb11a7bf4474e2f"}},{"code_sha256_prefix":"fff4dd3a3388381f","entry":"KernelMachines","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fff4dd3a3388381f"}},{"code_sha256_prefix":"7f5ae78b574db2df","entry":"NonEquivariantKernelMachine","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f5ae78b574db2df"}},{"code_sha256_prefix":"8a698dc7543122d9","entry":"kernel_index","repo":"sflippl/compositional-generalization","repo_kind":"official","path":"simple_compositional_tasks/models/kernel_machines.py","file_url":"https://github.com/sflippl/compositional-generalization/blob/HEAD/simple_compositional_tasks/models/kernel_machines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a698dc7543122d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}