{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pretraining-task-diversity-and-the-emergence-1","title":"Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression","arxiv_id":"2306.15063","date":"2023-06-26","proceeding":"NeurIPS 2023 11","authors":["Allan Raventós","Mansheej Paul","Feng Chen","Surya Ganguli"],"abstract":"Pretrained transformers exhibit the remarkable ability of in-context learning (ICL): they can learn tasks from just a few examples provided in the prompt without updating any weights. This raises a foundational question: can ICL solve fundamentally $\\textit{new}$ tasks that are very different from those seen during pretraining? To probe this question, we examine ICL's performance on linear regression while varying the diversity of tasks in the pretraining dataset. We empirically demonstrate a $\\textit{task diversity threshold}$ for the emergence of ICL. Below this threshold, the pretrained transformer cannot solve unseen regression tasks, instead behaving like a Bayesian estimator with the $\\textit{non-diverse pretraining task distribution}$ as the prior. Beyond this threshold, the transformer significantly outperforms this estimator; its behavior aligns with that of ridge regression, corresponding to a Gaussian prior over $\\textit{all tasks}$, including those not seen during pretraining. Thus, when pretrained on data with task diversity greater than the threshold, transformers $\\textit{can}$ optimally solve fundamentally new tasks in-context. Importantly, this capability hinges on it deviating from the Bayes optimal estimator with the pretraining distribution as the prior. This study also explores the effect of regularization, model capacity and task structure and underscores, in a concrete example, the critical role of task diversity, alongside data and model scale, in the emergence of ICL. Code is available at https://github.com/mansheej/icl-task-diversity.","url_abs":"https://arxiv.org/abs/2306.15063v2","url_pdf":"https://arxiv.org/pdf/2306.15063v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pretraining-task-diversity-and-the-emergence-1","repo_url":"https://github.com/mansheej/icl-task-diversity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.15063","atlas_url":"https://app.syntology.ai/?focus=2306.15063","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.15063"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mansheej/icl-task-diversity","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6b41eeafac23b0cc","entry":"get_optimizer_and_lr_schedule","repo":"mansheej/icl-task-diversity","repo_kind":"official","path":"icl/optim.py","file_url":"https://github.com/mansheej/icl-task-diversity/blob/HEAD/icl/optim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b41eeafac23b0cc"}},{"code_sha256_prefix":"bf9ee877432bf7f0","entry":"get_task_name","repo":"mansheej/icl-task-diversity","repo_kind":"official","path":"icl/tasks.py","file_url":"https://github.com/mansheej/icl-task-diversity/blob/HEAD/icl/tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf9ee877432bf7f0"}},{"code_sha256_prefix":"662cd4052f4fb4cd","entry":"mse","repo":"mansheej/icl-task-diversity","repo_kind":"official","path":"icl/evaluate.py","file_url":"https://github.com/mansheej/icl-task-diversity/blob/HEAD/icl/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"662cd4052f4fb4cd"}},{"code_sha256_prefix":"42850e2c30d04603","entry":"to_seq","repo":"mansheej/icl-task-diversity","repo_kind":"official","path":"icl/utils.py","file_url":"https://github.com/mansheej/icl-task-diversity/blob/HEAD/icl/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42850e2c30d04603"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}