{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/in-context-language-learning-arhitectures-and","title":"In-Context Language Learning: Architectures and Algorithms","arxiv_id":"2401.12973","date":"2024-01-23","proceeding":null,"authors":["Ekin Akyürek","Bailin Wang","Yoon Kim","Jacob Andreas"],"abstract":"Large-scale neural language models exhibit a remarkable capacity for in-context learning (ICL): they can infer novel functions from datasets provided as input. Most of our current understanding of when and how ICL arises comes from LMs trained on extremely simple learning problems like linear regression and associative recall. There remains a significant gap between these model problems and the \"real\" ICL exhibited by LMs trained on large text corpora, which involves not just retrieval and function approximation but free-form generation of language and other structured outputs. In this paper, we study ICL through the lens of a new family of model problems we term in context language learning (ICLL). In ICLL, LMs are presented with a set of strings from a formal language, and must generate additional strings from the same language. We focus on in-context learning of regular languages generated by random finite automata. We evaluate a diverse set of neural sequence models (including several RNNs, Transformers, and state-space model variants) on regular ICLL tasks, aiming to answer three questions: (1) Which model classes are empirically capable of ICLL? (2) What algorithmic solutions do successful models implement to perform ICLL? (3) What architectural changes can improve ICLL in less performant models? We first show that Transformers significantly outperform neural sequence models with recurrent or convolutional representations on ICLL tasks. Next, we provide evidence that their ability to do so relies on specialized \"n-gram heads\" (higher-order variants of induction heads) that compute input-conditional next-token distributions. Finally, we show that hard-wiring these heads into neural models improves performance not just on ICLL, but natural language modeling -- improving the perplexity of 340M-parameter models by up to 1.14 points (6.7%) on the SlimPajama dataset.","url_abs":"https://arxiv.org/abs/2401.12973v2","url_pdf":"https://arxiv.org/pdf/2401.12973v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"in-context-language-learning-arhitectures-and","repo_url":"https://github.com/berlino/seq_icl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"linear-regression","method_name":"Linear Regression"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.12973","atlas_url":"https://app.syntology.ai/?focus=2401.12973","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.12973"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/berlino/seq_icl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":8},"by_repo_kind":{"official":{"samples":20,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b7ffd8fb4517d412","entry":"create_flags","repo":"berlino/seq_icl","repo_kind":"official","path":"sweep.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7ffd8fb4517d412"}},{"code_sha256_prefix":"9e64222cc89cf42d","entry":"generate","repo":"berlino/seq_icl","repo_kind":"official","path":"generate.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e64222cc89cf42d"}},{"code_sha256_prefix":"51b7e5bf992a7ada","entry":"get_conditional_prob","repo":"berlino/seq_icl","repo_kind":"official","path":"ngram.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/ngram.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51b7e5bf992a7ada"}},{"code_sha256_prefix":"aa4ed806d118746a","entry":"get_config","repo":"berlino/seq_icl","repo_kind":"official","path":"sweep.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa4ed806d118746a"}},{"code_sha256_prefix":"9a83c3eaaa9c553c","entry":"get_dfa_states","repo":"berlino/seq_icl","repo_kind":"official","path":"samestateprobe.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/samestateprobe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a83c3eaaa9c553c"}},{"code_sha256_prefix":"ce9f5b21ad841769","entry":"get_sweep","repo":"berlino/seq_icl","repo_kind":"official","path":"sweep.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce9f5b21ad841769"}},{"code_sha256_prefix":"9f129256cb54eace","entry":"mask_transition_matrix","repo":"berlino/seq_icl","repo_kind":"official","path":"batched_baum_welch.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/batched_baum_welch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f129256cb54eace"}},{"code_sha256_prefix":"e71438a3ed6a9798","entry":"normalize","repo":"berlino/seq_icl","repo_kind":"official","path":"ngram.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/ngram.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e71438a3ed6a9798"}},{"code_sha256_prefix":"35218296da840eec","entry":"possible_states","repo":"berlino/seq_icl","repo_kind":"official","path":"batched_baum_welch.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/batched_baum_welch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35218296da840eec"}},{"code_sha256_prefix":"e13fb3b9ab87c890","entry":"read_hidden_states","repo":"berlino/seq_icl","repo_kind":"official","path":"samestateprobe.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/samestateprobe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e13fb3b9ab87c890"}},{"code_sha256_prefix":"dba3844475b7ec75","entry":"read_one","repo":"berlino/seq_icl","repo_kind":"official","path":"probe.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/probe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dba3844475b7ec75"}},{"code_sha256_prefix":"29c617b8c430aff7","entry":"train_everygram","repo":"berlino/seq_icl","repo_kind":"official","path":"ngram.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/ngram.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29c617b8c430aff7"}},{"code_sha256_prefix":"d079b2adb7787354","entry":"eval_dfa","repo":"berlino/seq_icl","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d079b2adb7787354"}},{"code_sha256_prefix":"7d16fa77eed96638","entry":"forward_backward","repo":"berlino/seq_icl","repo_kind":"official","path":"batched_baum_welch.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/batched_baum_welch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d16fa77eed96638"}},{"code_sha256_prefix":"e4d688cc5cf62263","entry":"get_results","repo":"berlino/seq_icl","repo_kind":"official","path":"samestateprobe.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/samestateprobe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4d688cc5cf62263"}},{"code_sha256_prefix":"7ba8c48c6e6661b3","entry":"get_transition_info","repo":"berlino/seq_icl","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ba8c48c6e6661b3"}},{"code_sha256_prefix":"5a82f62a07d935e2","entry":"get_uniform_probs","repo":"berlino/seq_icl","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a82f62a07d935e2"}},{"code_sha256_prefix":"22b5e068ce364282","entry":"prepare_video","repo":"berlino/seq_icl","repo_kind":"official","path":"make_video.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/make_video.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22b5e068ce364282"}},{"code_sha256_prefix":"544b7de38c79c09c","entry":"read_hidden_states","repo":"berlino/seq_icl","repo_kind":"official","path":"probe.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/probe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"544b7de38c79c09c"}},{"code_sha256_prefix":"e490d66f05b1420b","entry":"read_parallel","repo":"berlino/seq_icl","repo_kind":"official","path":"probe.py","file_url":"https://github.com/berlino/seq_icl/blob/HEAD/probe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e490d66f05b1420b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}