{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-18124","title":"TLoRA: Task-aware Low Rank Adaptation of Large Language Models","arxiv_id":"2604.18124","date":"2026-04-20","proceeding":null,"authors":["Weicheng Lin","Yi Zhang","Jiawei Dang","Liang-Jie Zhang"],"abstract":"Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA $A$ matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the $A$ matrix is frozen, and only the $B$ matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.","url_abs":"https://arxiv.org/abs/2604.18124","url_pdf":"https://arxiv.org/pdf/2604.18124","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.18124","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.18124"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Rambo-Yi/TLora","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"0fe9cf6c00ef56ea","entry":"is_number","repo":"Rambo-Yi/TLora","repo_kind":"found_in_text","path":"evaluate/math/eval_gsm8k.py","file_url":"https://github.com/Rambo-Yi/TLora/blob/HEAD/evaluate/math/eval_gsm8k.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe9cf6c00ef56ea"}},{"code_sha256_prefix":"0b14c648516c38a7","entry":"last_boxed_only_string","repo":"Rambo-Yi/TLora","repo_kind":"found_in_text","path":"evaluate/math/util.py","file_url":"https://github.com/Rambo-Yi/TLora/blob/HEAD/evaluate/math/util.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b14c648516c38a7"}},{"code_sha256_prefix":"f3bbe264b05aadd3","entry":"remove_boxed","repo":"Rambo-Yi/TLora","repo_kind":"found_in_text","path":"evaluate/math/eval_math.py","file_url":"https://github.com/Rambo-Yi/TLora/blob/HEAD/evaluate/math/eval_math.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3bbe264b05aadd3"}},{"code_sha256_prefix":"d49412b3a690206e","entry":"batch_data","repo":"Rambo-Yi/TLora","repo_kind":"found_in_text","path":"evaluate/code/gen_vllm.py","file_url":"https://github.com/Rambo-Yi/TLora/blob/HEAD/evaluate/code/gen_vllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d49412b3a690206e"}},{"code_sha256_prefix":"8720975b15d076e8","entry":"last_boxed_only","repo":"Rambo-Yi/TLora","repo_kind":"found_in_text","path":"evaluate/math/util.py","file_url":"https://github.com/Rambo-Yi/TLora/blob/HEAD/evaluate/math/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8720975b15d076e8"}},{"code_sha256_prefix":"0bbf876ced018c83","entry":"only_until_first_boxed_from_tokens","repo":"Rambo-Yi/TLora","repo_kind":"found_in_text","path":"evaluate/math/util.py","file_url":"https://github.com/Rambo-Yi/TLora/blob/HEAD/evaluate/math/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bbf876ced018c83"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}