{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-20730","title":"Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning","arxiv_id":"2605.20730","date":"2026-05-20","proceeding":null,"authors":["Jihoon Kwon","Jiwon Choi","Jy-yong Sohn"],"abstract":"In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks through demonstrations, yet it suffers from escalating inference costs as context length increases. While task vectors offer a promising alternative by compressing demonstrations into compact hidden-state representations, their quality has been evaluated only through downstream task accuracy. This indirect criterion provides limited insight into how to design more effective task vector extraction methods. In this paper, we posit that inference using task vectors should align their predictive distribution with that of ICL. To quantify this, we introduce $d_{\\text{NTP}}$, a metric that measures the discrepancy in next-token probabilities between task vector-based and ICL-based inference. Our empirical analysis reveals that $d_{\\text{NTP}}$ serves as a performance proxy, exhibiting a strong negative correlation with downstream accuracy. Motivated by this, we develop Linear Task Vector (LTV), a method designed to minimize $d_{\\text{NTP}}$ via a closed-form linear mapping that estimates demonstration effects through regression. Across eight classification benchmarks and five LLMs, LTV consistently outperforms existing task vector baselines, improving average accuracy by 9.2\\% while reducing inference latency. We further show that LTV outperforms the baselines on regression tasks. Moreover, we investigate the transferability of LTV across different model scales; an aspect that has remained nascent in task vector research. Specifically, we empirically show that task vectors from a larger model can enhance a smaller model's performance by 6.4\\%, suggesting a new utility for extracted task representations.","url_abs":"https://arxiv.org/abs/2605.20730","url_pdf":"https://arxiv.org/pdf/2605.20730","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.20730"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/HITsz-TMG/ICL-State-Vector","reach":{"status":"ok"}},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/LzVv123456/I2CL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_honours":1,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":7,"ran":6,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"7fa4c0d4e9cee1af","entry":"get_data_layers","repo":"HITsz-TMG/ICL-State-Vector","repo_kind":"found_in_text","path":"selected_layers.py","file_url":"https://github.com/HITsz-TMG/ICL-State-Vector/blob/HEAD/selected_layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fa4c0d4e9cee1af"}},{"code_sha256_prefix":"dc25dfef6f90b386","entry":"metric_average","repo":"HITsz-TMG/ICL-State-Vector","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/HITsz-TMG/ICL-State-Vector/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc25dfef6f90b386"}},{"code_sha256_prefix":"85ac527a9dce9de6","entry":"sea_born","repo":"HITsz-TMG/ICL-State-Vector","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/HITsz-TMG/ICL-State-Vector/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85ac527a9dce9de6"}},{"code_sha256_prefix":"f4d8b3796ddaccdf","entry":"shuffle_train","repo":"HITsz-TMG/ICL-State-Vector","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/HITsz-TMG/ICL-State-Vector/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f4d8b3796ddaccdf"}},{"code_sha256_prefix":"3b34facc51914c48","entry":"str2list","repo":"HITsz-TMG/ICL-State-Vector","repo_kind":"found_in_text","path":"state_vector_optimize.py","file_url":"https://github.com/HITsz-TMG/ICL-State-Vector/blob/HEAD/state_vector_optimize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b34facc51914c48"}},{"code_sha256_prefix":"405289a97499d9cc","entry":"target_layer_selection","repo":"LzVv123456/I2CL","repo_kind":"found_in_text","path":"run_task_vector.py","file_url":"https://github.com/LzVv123456/I2CL/blob/HEAD/run_task_vector.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"405289a97499d9cc"}},{"code_sha256_prefix":"f5d8f58d3af60f15","entry":"get_optimizers","repo":"HITsz-TMG/ICL-State-Vector","repo_kind":"found_in_text","path":"optimizer.py","file_url":"https://github.com/HITsz-TMG/ICL-State-Vector/blob/HEAD/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5d8f58d3af60f15"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}