{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-distributional-properties-drive-emergent","title":"Data Distributional Properties Drive Emergent In-Context Learning in Transformers","arxiv_id":"2205.05055","date":"2022-04-22","proceeding":null,"authors":["Stephanie C. Y. Chan","Adam Santoro","Andrew K. Lampinen","Jane X. Wang","Aaditya Singh","Pierre H. Richemond","Jay McClelland","Felix Hill"],"abstract":"Large transformer-based models are able to perform in-context few-shot learning, without being explicitly trained for it. This observation raises the question: what aspects of the training regime lead to this emergent behavior? Here, we show that this behavior is driven by the distributions of the training data itself. In-context learning emerges when the training data exhibits particular distributional properties such as burstiness (items appear in clusters rather than being uniformly distributed over time) and having large numbers of rarely occurring classes. In-context learning also emerges more strongly when item meanings or interpretations are dynamic rather than fixed. These properties are exemplified by natural language, but are also inherent to naturalistic data in a wide range of other domains. They also depart significantly from the uniform, i.i.d. training distributions typically used for standard supervised learning. In our initial experiments, we found that in-context learning traded off against more conventional weight-based learning, and models were unable to achieve both simultaneously. However, our later experiments uncovered that the two modes of learning could co-exist in a single model when it was trained on data following a skewed Zipfian distribution -- another common property of naturalistic data, including language. In further experiments, we found that naturalistic data distributions were only able to elicit in-context learning in transformers, and not in recurrent models. In sum, our findings indicate how the transformer architecture works together with particular properties of the training data to drive the intriguing emergent in-context learning behaviour of large language models, and how future work might encourage both in-context and in-weights learning in domains beyond language.","url_abs":"https://arxiv.org/abs/2205.05055v6","url_pdf":"https://arxiv.org/pdf/2205.05055v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-distributional-properties-drive-emergent","repo_url":"https://github.com/deepmind/emergent_in_context_learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"data-distributional-properties-drive-emergent","repo_url":"https://github.com/aadityasingh/icl-dynamics","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"data-distributional-properties-drive-emergent","repo_url":"https://github.com/aadityasingh/icl-transience","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}},{"paper_slug":"data-distributional-properties-drive-emergent","repo_url":"https://github.com/google-deepmind/emergent_in_context_learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2205.05055","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.05055"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/emergent_in_context_learning","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/emergent_in_context_learning","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aadityasingh/icl-transience","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aadityasingh/icl-dynamics","reach":null}],"summary":{"ran_fixture":1,"ran_honours":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"369d990a59246f7f","entry":"apply_rope","repo":"aadityasingh/icl-dynamics","repo_kind":"listed","path":"models.py","file_url":"https://github.com/aadityasingh/icl-dynamics/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"369d990a59246f7f"}},{"code_sha256_prefix":"6c3305fbafc272c8","entry":"create_sinusoidal_positions","repo":"aadityasingh/icl-dynamics","repo_kind":"listed","path":"models.py","file_url":"https://github.com/aadityasingh/icl-dynamics/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c3305fbafc272c8"}},{"code_sha256_prefix":"45325303dac3d895","entry":"make_cache_assign","repo":"aadityasingh/icl-dynamics","repo_kind":"listed","path":"models.py","file_url":"https://github.com/aadityasingh/icl-dynamics/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45325303dac3d895"}},{"code_sha256_prefix":"652a0680a67dcd73","entry":"conv1","repo":"deepmind/emergent_in_context_learning","repo_kind":"official","path":"modules/transformer_core.py","file_url":"https://github.com/deepmind/emergent_in_context_learning/blob/HEAD/modules/transformer_core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"652a0680a67dcd73"}},{"code_sha256_prefix":"4821ab9352b9a1c1","entry":"get_pos_start","repo":"deepmind/emergent_in_context_learning","repo_kind":"official","path":"modules/transformer_core.py","file_url":"https://github.com/deepmind/emergent_in_context_learning/blob/HEAD/modules/transformer_core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4821ab9352b9a1c1"}},{"code_sha256_prefix":"5d83309868b4e1a6","entry":"layer_norm","repo":"deepmind/emergent_in_context_learning","repo_kind":"official","path":"modules/transformer_core.py","file_url":"https://github.com/deepmind/emergent_in_context_learning/blob/HEAD/modules/transformer_core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d83309868b4e1a6"}},{"code_sha256_prefix":"2fa569a9eb572a35","entry":"prepare_seqs_for_transformer","repo":"deepmind/emergent_in_context_learning","repo_kind":"official","path":"datasets/utils.py","file_url":"https://github.com/deepmind/emergent_in_context_learning/blob/HEAD/datasets/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fa569a9eb572a35"}},{"code_sha256_prefix":"34959b5d69981166","entry":"reduce_fn","repo":"deepmind/emergent_in_context_learning","repo_kind":"official","path":"modules/losses.py","file_url":"https://github.com/deepmind/emergent_in_context_learning/blob/HEAD/modules/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34959b5d69981166"}},{"code_sha256_prefix":"c8b1743b116dcfc8","entry":"softmax_cross_entropy","repo":"deepmind/emergent_in_context_learning","repo_kind":"official","path":"modules/losses.py","file_url":"https://github.com/deepmind/emergent_in_context_learning/blob/HEAD/modules/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8b1743b116dcfc8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}