{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/function-vectors-in-large-language-models","title":"Function Vectors in Large Language Models","arxiv_id":"2310.15213","date":"2023-10-23","proceeding":null,"authors":["Eric Todd","Millicent L. Li","Arnab Sen Sharma","Aaron Mueller","Byron C. Wallace","David Bau"],"abstract":"We report the presence of a simple neural mechanism that represents an input-output function as a vector within autoregressive transformer language models (LMs). Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which we call a function vector (FV). FVs are robust to changes in context, i.e., they trigger execution of the task on inputs such as zero-shot and natural text settings that do not resemble the ICL contexts from which they are collected. We test FVs across a range of tasks, models, and layers and find strong causal effects across settings in middle layers. We investigate the internal structure of FVs and find while that they often contain information that encodes the output space of the function, this information alone is not sufficient to reconstruct an FV. Finally, we test semantic vector composition in FVs, and find that to some extent they can be summed to create vectors that trigger new complex tasks. Our findings show that compact, causal internal vector representations of function abstractions can be explicitly extracted from LLMs. Our code and data are available at https://functions.baulab.info.","url_abs":"https://arxiv.org/abs/2310.15213v2","url_pdf":"https://arxiv.org/pdf/2310.15213v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"function-vectors-in-large-language-models","repo_url":"https://github.com/ericwtodd/function_vectors","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.15213","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.15213"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ericwtodd/function_vectors","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"listed":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a9853ae06ebc5529","entry":"compute_best_token_rank","repo":"ericwtodd/function_vectors","repo_kind":"listed","path":"src/utils/eval_utils.py","file_url":"https://github.com/ericwtodd/function_vectors/blob/HEAD/src/utils/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9853ae06ebc5529"}},{"code_sha256_prefix":"06461c0c4a4708ce","entry":"compute_individual_token_rank","repo":"ericwtodd/function_vectors","repo_kind":"listed","path":"src/utils/eval_utils.py","file_url":"https://github.com/ericwtodd/function_vectors/blob/HEAD/src/utils/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06461c0c4a4708ce"}},{"code_sha256_prefix":"6351069e7766eb7a","entry":"compute_top_k_accuracy","repo":"ericwtodd/function_vectors","repo_kind":"listed","path":"src/utils/eval_utils.py","file_url":"https://github.com/ericwtodd/function_vectors/blob/HEAD/src/utils/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6351069e7766eb7a"}},{"code_sha256_prefix":"8953f9ef90cf6afd","entry":"get_module","repo":"ericwtodd/function_vectors","repo_kind":"listed","path":"src/utils/intervention_utils.py","file_url":"https://github.com/ericwtodd/function_vectors/blob/HEAD/src/utils/intervention_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8953f9ef90cf6afd"}},{"code_sha256_prefix":"0095bc7827bfbc5e","entry":"add_function_vector","repo":"ericwtodd/function_vectors","repo_kind":"listed","path":"src/utils/intervention_utils.py","file_url":"https://github.com/ericwtodd/function_vectors/blob/HEAD/src/utils/intervention_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0095bc7827bfbc5e"}},{"code_sha256_prefix":"3eba226ef375d48e","entry":"load_gpt_model_and_tokenizer","repo":"ericwtodd/function_vectors","repo_kind":"listed","path":"src/utils/model_utils.py","file_url":"https://github.com/ericwtodd/function_vectors/blob/HEAD/src/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3eba226ef375d48e"}},{"code_sha256_prefix":"a97b73b55d820f6a","entry":"optim_loop","repo":"ericwtodd/function_vectors","repo_kind":"listed","path":"src/vocab_reconstruction.py","file_url":"https://github.com/ericwtodd/function_vectors/blob/HEAD/src/vocab_reconstruction.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a97b73b55d820f6a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}