{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-do-large-language-models-learn-in-context","title":"How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning","arxiv_id":"2402.02872","date":"2024-02-05","proceeding":null,"authors":["Zeping Yu","Sophia Ananiadou"],"abstract":"We investigate the mechanism of in-context learning (ICL) on sentence classification tasks with semantically-unrelated labels (\"foo\"/\"bar\"). We find intervening in only 1\\% heads (named \"in-context heads\") significantly affects ICL accuracy from 87.6\\% to 24.4\\%. To understand this phenomenon, we analyze the value-output vectors in these heads and discover that the vectors at each label position contain substantial information about the corresponding labels. Furthermore, we observe that the prediction shift from \"foo\" to \"bar\" is due to the respective reduction and increase in these heads' attention scores at \"foo\" and \"bar\" positions. Therefore, we propose a hypothesis for ICL: in in-context heads, the value-output matrices extract label features, while the query-key matrices compute the similarity between the features at the last position and those at each label position. The query and key matrices can be considered as two towers that learn the similarity metric between the last position's features and each demonstration at label positions. Using this hypothesis, we explain the majority label bias and recency bias in ICL and propose two methods to reduce these biases by 22\\% and 17\\%, respectively.","url_abs":"https://arxiv.org/abs/2402.02872v3","url_pdf":"https://arxiv.org/pdf/2402.02872v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-do-large-language-models-learn-in-context","repo_url":"https://github.com/zepingyu0512/in-context-mechanism","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"how-do-large-language-models-learn-in-context","repo_url":"https://github.com/zepingyu0512/neuron-attribution","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":null,"task_name":"Position"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-classification","task_name":"Sentence Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.02872","atlas_url":"https://app.syntology.ai/?focus=2402.02872","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02872"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zepingyu0512/neuron-attribution","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zepingyu0512/in-context-mechanism","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"zepingyu0512/in-context-mechanism","repo_kind":"official","path":"modeling_llama.py","file_url":"https://github.com/zepingyu0512/in-context-mechanism/blob/HEAD/modeling_llama.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"zepingyu0512/in-context-mechanism","repo_kind":"official","path":"modeling_llama.py","file_url":"https://github.com/zepingyu0512/in-context-mechanism/blob/HEAD/modeling_llama.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"zepingyu0512/in-context-mechanism","repo_kind":"official","path":"modeling_llama.py","file_url":"https://github.com/zepingyu0512/in-context-mechanism/blob/HEAD/modeling_llama.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}