{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-models-can-exploit-cross-task-in","title":"Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks","arxiv_id":"2405.10548","date":"2024-05-17","proceeding":null,"authors":["Anwoy Chatterjee","Eshaan Tanwar","Subhabrata Dutta","Tanmoy Chakraborty"],"abstract":"Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning (ICL) capabilities. Automated assistants based on LLMs are gaining popularity; however, adapting them to novel tasks is still challenging. While colossal models excel in zero-shot performance, their computational demands limit widespread use, and smaller language models struggle without context. This paper investigates whether LLMs can generalize from labeled examples of predefined tasks to novel tasks. Drawing inspiration from biological neurons and the mechanistic interpretation of the Transformer architecture, we explore the potential for information sharing across tasks. We design a cross-task prompting setup with three LLMs and show that LLMs achieve significant performance improvements despite no examples from the target task in the context. Cross-task prompting leads to a remarkable performance boost of 107% for LLaMA-2 7B, 18.6% for LLaMA-2 13B, and 3.2% for GPT 3.5 on average over zero-shot prompting, and performs comparable to standard in-context learning. The effectiveness of generating pseudo-labels for in-task examples is demonstrated, and our analyses reveal a strong correlation between the effect of cross-task examples and model activation similarities in source and target input tokens. This paper offers a first-of-its-kind exploration of LLMs' ability to solve novel tasks based on contextual signals from different task examples.","url_abs":"https://arxiv.org/abs/2405.10548v3","url_pdf":"https://arxiv.org/pdf/2405.10548v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-models-can-exploit-cross-task-in","repo_url":"https://github.com/c-anwoy/cross-task-icl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.10548","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.10548"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/C-anwoy/Cross-Task-ICL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"00eba4d4d2e34691","entry":"call_model","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"LLAMA-abalations-random.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/LLAMA-abalations-random.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00eba4d4d2e34691"}},{"code_sha256_prefix":"64ac3a4563570515","entry":"create_embedding","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"utils/llm.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/utils/llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"64ac3a4563570515"}},{"code_sha256_prefix":"5de5f3647524eb25","entry":"get_combined_source_dataset","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"LLAMA-abalations.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/LLAMA-abalations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5de5f3647524eb25"}},{"code_sha256_prefix":"909d7f2bdccf6dfe","entry":"get_llm_emb","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"utils/llm.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/utils/llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"909d7f2bdccf6dfe"}},{"code_sha256_prefix":"982e3655e9695d47","entry":"load_dataset","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"982e3655e9695d47"}},{"code_sha256_prefix":"e401d40128e5c42a","entry":"load_prompts","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e401d40128e5c42a"}},{"code_sha256_prefix":"64e4f9b4c5d53a6b","entry":"make_prompt","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"LLAMA-psudo.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/LLAMA-psudo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"64e4f9b4c5d53a6b"}},{"code_sha256_prefix":"83f6c004d81e3857","entry":"force_decode","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"force_decode.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/force_decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83f6c004d81e3857"}},{"code_sha256_prefix":"fb8470d7386eb6b5","entry":"get_combined_source_dataset","repo":"C-anwoy/Cross-Task-ICL","repo_kind":"official","path":"LLAMA-abalations-random.py","file_url":"https://github.com/C-anwoy/Cross-Task-ICL/blob/HEAD/LLAMA-abalations-random.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb8470d7386eb6b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}