{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformers-learn-in-context-by-gradient","title":"Transformers learn in-context by gradient descent","arxiv_id":"2212.07677","date":"2022-12-15","proceeding":null,"authors":["Johannes von Oswald","Eyvind Niklasson","Ettore Randazzo","João Sacramento","Alexander Mordvintsev","Andrey Zhmoginov","Max Vladymyrov"],"abstract":"At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss. Motivated by that construction, we show empirically that when training self-attention-only Transformers on simple regression tasks either the models learned by GD and Transformers show great similarity or, remarkably, the weights found by optimization match the construction. Thus we show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass. This allows us, at least in the domain of regression problems, to mechanistically understand the inner workings of in-context learning in optimized Transformers. Building on this insight, we furthermore identify how Transformers surpass the performance of plain gradient descent by learning an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks. Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers. Code to reproduce the experiments can be found at https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .","url_abs":"https://arxiv.org/abs/2212.07677v2","url_pdf":"https://arxiv.org/pdf/2212.07677v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformers-learn-in-context-by-gradient","repo_url":"https://github.com/google-research/self-organising-systems","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"transformers-learn-in-context-by-gradient","repo_url":"https://github.com/RobvanGastel/meta-in-context-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.07677","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.07677"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/self-organising-systems","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RobvanGastel/meta-in-context-learning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":16},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1},"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"15182f943143a6c8","entry":"arrayContains","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/utils.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15182f943143a6c8"}},{"code_sha256_prefix":"32143cfb8422ba7f","entry":"augment_tasks","repo":"RobvanGastel/meta-in-context-learning","repo_kind":"listed","path":"meta_icl/transforms.py","file_url":"https://github.com/RobvanGastel/meta-in-context-learning/blob/HEAD/meta_icl/transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32143cfb8422ba7f"}},{"code_sha256_prefix":"75c456f521df57df","entry":"clip_residual","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/agent_logic.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/agent_logic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75c456f521df57df"}},{"code_sha256_prefix":"824bd8da9ddf7f92","entry":"conditional_update","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/utils.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"824bd8da9ddf7f92"}},{"code_sha256_prefix":"ca2ba6b3ce6e7e18","entry":"create_reg_dataset","repo":"RobvanGastel/meta-in-context-learning","repo_kind":"listed","path":"meta_icl/data.py","file_url":"https://github.com/RobvanGastel/meta-in-context-learning/blob/HEAD/meta_icl/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca2ba6b3ce6e7e18"}},{"code_sha256_prefix":"38f91bb22eeeaa13","entry":"imencode","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/display_utils.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/display_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38f91bb22eeeaa13"}},{"code_sha256_prefix":"17c1d14bf36b249a","entry":"make_empty_agent_state_cell","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/env_logic.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/env_logic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17c1d14bf36b249a"}},{"code_sha256_prefix":"15c4c038d8a867cf","entry":"make_empty_exclusive_op_cell","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/env_logic.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/env_logic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15c4c038d8a867cf"}},{"code_sha256_prefix":"8c04902f0945c434","entry":"make_empty_grid_agent_id","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/environments.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/environments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c04902f0945c434"}},{"code_sha256_prefix":"372a22e7a0877aa4","entry":"make_empty_grid_state","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/environments.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/environments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"372a22e7a0877aa4"}},{"code_sha256_prefix":"520aa1da21b7b789","entry":"make_empty_grid_type","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/environments.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/environments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"520aa1da21b7b789"}},{"code_sha256_prefix":"b8693f5392a22b13","entry":"make_empty_upd_state","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/env_logic.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/env_logic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8693f5392a22b13"}},{"code_sha256_prefix":"eebeba31f9da6479","entry":"np2pil","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/display_utils.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/display_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eebeba31f9da6479"}},{"code_sha256_prefix":"925adf7210923459","entry":"sample_regression_dataset","repo":"RobvanGastel/meta-in-context-learning","repo_kind":"listed","path":"meta_icl/data.py","file_url":"https://github.com/RobvanGastel/meta-in-context-learning/blob/HEAD/meta_icl/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"925adf7210923459"}},{"code_sha256_prefix":"597aadacf31a2d26","entry":"split_2d","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/utils.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"597aadacf31a2d26"}},{"code_sha256_prefix":"16e85b6c82b85822","entry":"valuenoise1d","repo":"google-research/self-organising-systems","repo_kind":"official","path":"self_organising_systems/biomakerca/mutators.py","file_url":"https://github.com/google-research/self-organising-systems/blob/HEAD/self_organising_systems/biomakerca/mutators.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16e85b6c82b85822"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}