{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-05676","title":"Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning","arxiv_id":"2605.05676","date":"2026-05-07","proceeding":"ICML","authors":["Bing Wang","Ximing Li","Changchun Li","Jinjin Chi","Gang Niu","Masashi Sugiyama"],"abstract":"Recently, the prominent performance of large language models (LLMs) has been largely driven by multi-task instruct-tuning. Unfortunately, this training paradigm suffers from a key issue, named cross-task interference, due to conflicting gradients over shared parameters among different tasks. Some previous methods mitigate this issue by isolating task-specific parameters, e.g., task-specific neuron selection and mixture-of-experts. In this paper, we empirically reveal that the cross-task interference still exists for the existing solutions because of many parameters also shared by different tasks, and accordingly, we propose a novel solution, namely Basic Abilities Decomposition for multi-task Instruct-Tuning (BADIT). Specifically, we empirically find that certain parameters are consistently co-activated, and that co-activated parameters naturally organize into base groups. This motivates us to analogize that LLMs encode several orthogonal basic abilities, and that any task can be represented as a linear combination of these abilities. Accordingly, we propose BADIT that decomposes LLM parameters into orthogonal high-singular-value LoRA experts representing basic abilities, and dynamically enforces their orthogonality during training via spherical clustering of rank-1 components. We conduct extensive experiments on the SuperNI benchmark with 6 LLMs, and empirical results demonstrate that BADIT can outperform SOTA methods and mitigate the degree of cross-task interference.","url_abs":"https://arxiv.org/abs/2605.05676","url_pdf":"https://arxiv.org/pdf/2605.05676","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.05676","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.05676"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wangbing1416/BADIT","reach":null}],"summary":{"ran":3,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":11,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"89a7988d2785490d","entry":"PromptEmbedding","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89a7988d2785490d"}},{"code_sha256_prefix":"56d203f88b5ddcf2","entry":"PromptTuningConfig","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56d203f88b5ddcf2"}},{"code_sha256_prefix":"9f58161eb163457c","entry":"check_deepspeed_zero3_enabled","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f58161eb163457c"}},{"code_sha256_prefix":"c3e6580bf962762c","entry":"MultitaskPromptEmbedding","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3e6580bf962762c"}},{"code_sha256_prefix":"eaa167429c2471d2","entry":"MultitaskPromptTuningConfig","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eaa167429c2471d2"}},{"code_sha256_prefix":"c5d01d0685f2833e","entry":"MultitaskPromptTuningInit","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5d01d0685f2833e"}},{"code_sha256_prefix":"1bfb5c9ada928004","entry":"PeftType","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1bfb5c9ada928004"}},{"code_sha256_prefix":"e439108711e4b1c5","entry":"PromptTuningInit","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e439108711e4b1c5"}},{"code_sha256_prefix":"ac135e22aaee5c90","entry":"TaskType","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac135e22aaee5c90"}},{"code_sha256_prefix":"f810ca075da78d33","entry":"gather_params_ctx","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f810ca075da78d33"}},{"code_sha256_prefix":"922d9b65d9f360e7","entry":"torch_load","repo":"wangbing1416/BADIT","repo_kind":"found_in_text","path":"peft/tuners/multitask_prompt_tuning/model.py","file_url":"https://github.com/wangbing1416/BADIT/blob/HEAD/peft/tuners/multitask_prompt_tuning/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"922d9b65d9f360e7"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}