{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-can-transformers-ground-and-compose","title":"When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks","arxiv_id":"2210.12786","date":"2022-10-23","proceeding":null,"authors":["Ankur Sikarwar","Arkil Patel","Navin Goyal"],"abstract":"Humans can reason compositionally whilst grounding language utterances to the real world. Recent benchmarks like ReaSCAN use navigation tasks grounded in a grid world to assess whether neural models exhibit similar capabilities. In this work, we present a simple transformer-based model that outperforms specialized architectures on ReaSCAN and a modified version of gSCAN. On analyzing the task, we find that identifying the target location in the grid world is the main challenge for the models. Furthermore, we show that a particular split in ReaSCAN, which tests depth generalization, is unfair. On an amended version of this split, we show that transformers can generalize to deeper input structures. Finally, we design a simpler grounded compositional generalization task, RefEx, to investigate how transformers reason compositionally. We show that a single self-attention layer with a single head generalizes to novel combinations of object attributes. Moreover, we derive a precise mathematical construction of the transformer's computations from the learned network. Overall, we provide valuable insights about the grounded compositional generalization task and the behaviour of transformers on it, which would be useful for researchers working in this area.","url_abs":"https://arxiv.org/abs/2210.12786v2","url_pdf":"https://arxiv.org/pdf/2210.12786v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-can-transformers-ground-and-compose","repo_url":"https://github.com/ankursikarwar/grounded-compositional-generalization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"compositional-generalization-avg","task_name":"Compositional Generalization (AVG)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.12786","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.12786"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ankursikarwar/grounded-compositional-generalization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8630adb451905844","entry":"calc_target_accuracy","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_target.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_target.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8630adb451905844"}},{"code_sha256_prefix":"03fb912963c88308","entry":"greedy_decode_batch_dual_mcnn","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_mcnn.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_mcnn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03fb912963c88308"}},{"code_sha256_prefix":"d75fc675031d5c92","entry":"train_one_epoch_mcnn","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_mcnn.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_mcnn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d75fc675031d5c92"}},{"code_sha256_prefix":"2a1a8ca2202a8076","entry":"train_one_epoch_target","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_target.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_target.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a1a8ca2202a8076"}},{"code_sha256_prefix":"f9e94beba0354b20","entry":"train_one_epoch_target_simple","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_target_simple.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_target_simple.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9e94beba0354b20"}},{"code_sha256_prefix":"5094ea97fa470b20","entry":"validate_one_epoch_mcnn","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_mcnn.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_mcnn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5094ea97fa470b20"}},{"code_sha256_prefix":"1a805e8b4534d128","entry":"validate_one_epoch_target","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_target.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_target.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a805e8b4534d128"}},{"code_sha256_prefix":"958c6f494632868c","entry":"validate_one_epoch_target_simple","repo":"ankursikarwar/grounded-compositional-generalization","repo_kind":"official","path":"reascan/src/train_target_simple.py","file_url":"https://github.com/ankursikarwar/grounded-compositional-generalization/blob/HEAD/reascan/src/train_target_simple.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"958c6f494632868c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}