{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/strategy-coopetition-explains-the-emergence","title":"Strategy Coopetition Explains the Emergence and Transience of In-Context Learning","arxiv_id":"2503.05631","date":"2025-03-07","proceeding":null,"authors":["Aaditya K. Singh","Ted Moskovitz","Sara Dragutinovic","Felix Hill","Stephanie C. Y. Chan","Andrew M. Saxe"],"abstract":"In-context learning (ICL) is a powerful ability that emerges in transformer models, enabling them to learn from context without weight updates. Recent work has established emergent ICL as a transient phenomenon that can sometimes disappear after long training times. In this work, we sought a mechanistic understanding of these transient dynamics. Firstly, we find that, after the disappearance of ICL, the asymptotic strategy is a remarkable hybrid between in-weights and in-context learning, which we term \"context-constrained in-weights learning\" (CIWL). CIWL is in competition with ICL, and eventually replaces it as the dominant strategy of the model (thus leading to ICL transience). However, we also find that the two competing strategies actually share sub-circuits, which gives rise to cooperative dynamics as well. For example, in our setup, ICL is unable to emerge quickly on its own, and can only be enabled through the simultaneous slow development of asymptotic CIWL. CIWL thus both cooperates and competes with ICL, a phenomenon we term \"strategy coopetition.\" We propose a minimal mathematical model that reproduces these key dynamics and interactions. Informed by this model, we were able to identify a setup where ICL is truly emergent and persistent.","url_abs":"https://arxiv.org/abs/2503.05631v2","url_pdf":"https://arxiv.org/pdf/2503.05631v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"strategy-coopetition-explains-the-emergence","repo_url":"https://github.com/aadityasingh/icl-dynamics","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"strategy-coopetition-explains-the-emergence","repo_url":"https://github.com/aadityasingh/icl-transience","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2503.05631","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.05631"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aadityasingh/icl-transience","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aadityasingh/icl-dynamics","reach":null}],"summary":{"ran_fixture":1,"ran_honours":2,"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"369d990a59246f7f","entry":"apply_rope","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"369d990a59246f7f"}},{"code_sha256_prefix":"6c3305fbafc272c8","entry":"create_sinusoidal_positions","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"6c3305fbafc272c8"}},{"code_sha256_prefix":"e17dd5575df330fc","entry":"make_3_loss_fn","repo":"aadityasingh/icl-dynamics","repo_kind":"official","path":"coopetition_model_solver.py","file_url":"https://github.com/aadityasingh/icl-dynamics/blob/HEAD/coopetition_model_solver.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e17dd5575df330fc"}},{"code_sha256_prefix":"b3c5d753c5587ef8","entry":"make_4share2_loss_fn","repo":"aadityasingh/icl-dynamics","repo_kind":"official","path":"coopetition_model_solver.py","file_url":"https://github.com/aadityasingh/icl-dynamics/blob/HEAD/coopetition_model_solver.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3c5d753c5587ef8"}},{"code_sha256_prefix":"45325303dac3d895","entry":"make_cache_assign","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"45325303dac3d895"}},{"code_sha256_prefix":"164839259956b235","entry":"outer_product_loss","repo":"aadityasingh/icl-dynamics","repo_kind":"official","path":"coopetition_model_solver.py","file_url":"https://github.com/aadityasingh/icl-dynamics/blob/HEAD/coopetition_model_solver.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"164839259956b235"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}