{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/taia-large-language-models-are-out-of","title":"TAIA: Large Language Models are Out-of-Distribution Data Learners","arxiv_id":"2405.20192","date":"2024-05-30","proceeding":null,"authors":["Shuyang Jiang","Yusheng Liao","Ya zhang","Yanfeng Wang","Yu Wang"],"abstract":"Fine-tuning on task-specific question-answer pairs is a predominant method for enhancing the performance of instruction-tuned large language models (LLMs) on downstream tasks. However, in certain specialized domains, such as healthcare or harmless content generation, it is nearly impossible to obtain a large volume of high-quality data that matches the downstream distribution. To improve the performance of LLMs in data-scarce domains with domain-mismatched data, we re-evaluated the Transformer architecture and discovered that not all parameter updates during fine-tuning contribute positively to downstream performance. Our analysis reveals that within the self-attention and feed-forward networks, only the fine-tuned attention parameters are particularly beneficial when the training set's distribution does not fully align with the test set. Based on this insight, we propose an effective inference-time intervention method: Training All parameters but Inferring with only Attention (\\trainallInfAttn). We empirically validate \\trainallInfAttn using two general instruction-tuning datasets and evaluate it on seven downstream tasks involving math, reasoning, and knowledge understanding across LLMs of different parameter sizes and fine-tuning techniques. Our comprehensive experiments demonstrate that \\trainallInfAttn achieves superior improvements compared to both the fully fine-tuned model and the base model in most scenarios, with significant performance gains. The high tolerance of \\trainallInfAttn to data mismatches makes it resistant to jailbreaking tuning and enhances specialized tasks using general data. Code is available in \\url{https://github.com/pixas/TAIA_LLM}.","url_abs":"https://arxiv.org/abs/2405.20192v2","url_pdf":"https://arxiv.org/pdf/2405.20192v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"taia-large-language-models-are-out-of","repo_url":"https://github.com/pixas/TAIA_LLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.20192","atlas_url":"https://app.syntology.ai/?focus=2405.20192","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20192"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/pixas/TAIA_LLM","reach":null}],"summary":{"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"bf3460a19530da22","entry":"BaseModelOutputWithPastLogitLoss","repo":"pixas/TAIA_LLM","repo_kind":"official","path":"taia/model/modeling_molora_llama.py","file_url":"https://github.com/pixas/TAIA_LLM/blob/HEAD/taia/model/modeling_molora_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf3460a19530da22"}},{"code_sha256_prefix":"2aee7c286eb10fe8","entry":"MoLoRALlamaMLP","repo":"pixas/TAIA_LLM","repo_kind":"official","path":"taia/model/modeling_molora_llama.py","file_url":"https://github.com/pixas/TAIA_LLM/blob/HEAD/taia/model/modeling_molora_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2aee7c286eb10fe8"}},{"code_sha256_prefix":"700af2fc8dd6876b","entry":"MoLoRALlamaDecoderLayer","repo":"pixas/TAIA_LLM","repo_kind":"official","path":"taia/model/modeling_molora_llama.py","file_url":"https://github.com/pixas/TAIA_LLM/blob/HEAD/taia/model/modeling_molora_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"700af2fc8dd6876b"}},{"code_sha256_prefix":"c05245b47150fcc7","entry":"MoLoRALlamaForCausalLM","repo":"pixas/TAIA_LLM","repo_kind":"official","path":"taia/model/modeling_molora_llama.py","file_url":"https://github.com/pixas/TAIA_LLM/blob/HEAD/taia/model/modeling_molora_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c05245b47150fcc7"}},{"code_sha256_prefix":"bfee72985d8aa2e4","entry":"MoLoRALlamaModel","repo":"pixas/TAIA_LLM","repo_kind":"official","path":"taia/model/modeling_molora_llama.py","file_url":"https://github.com/pixas/TAIA_LLM/blob/HEAD/taia/model/modeling_molora_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfee72985d8aa2e4"}},{"code_sha256_prefix":"0a1c7b7fd806c8f9","entry":"calc_attention_fluctuation","repo":"pixas/taia_llm","repo_kind":"official","path":"taia/model/modeling_molora_llama.py","file_url":"https://github.com/pixas/taia_llm/blob/HEAD/taia/model/modeling_molora_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a1c7b7fd806c8f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}