{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parameter-efficient-multi-task-model-fusion","title":"Parameter Efficient Multi-task Model Fusion with Partial Linearization","arxiv_id":"2310.04742","date":"2023-10-07","proceeding":null,"authors":["Anke Tang","Li Shen","Yong Luo","Yibing Zhan","Han Hu","Bo Du","Yixin Chen","DaCheng Tao"],"abstract":"Large pre-trained models have enabled significant advances in machine learning and served as foundation components. Model fusion methods, such as task arithmetic, have been proven to be powerful and scalable to incorporate fine-tuned weights from different tasks into a multi-task model. However, efficiently fine-tuning large pre-trained models on multiple downstream tasks remains challenging, leading to inefficient multi-task model fusion. In this work, we propose a novel method to improve multi-task fusion for parameter-efficient fine-tuning techniques like LoRA fine-tuning. Specifically, our approach partially linearizes only the adapter modules and applies task arithmetic over the linearized adapters. This allows us to leverage the the advantages of model fusion over linearized fine-tuning, while still performing fine-tuning and inference efficiently. We demonstrate that our partial linearization technique enables a more effective fusion of multiple tasks into a single model, outperforming standard adapter tuning and task arithmetic alone. Experimental results demonstrate the capabilities of our proposed partial linearization technique to effectively construct unified multi-task models via the fusion of fine-tuned task vectors. We evaluate performance over an increasing number of tasks and find that our approach outperforms standard parameter-efficient fine-tuning techniques. The results highlight the benefits of partial linearization for scalable and efficient multi-task model fusion. The code is available at https://github.com/tanganke/peta","url_abs":"https://arxiv.org/abs/2310.04742v3","url_pdf":"https://arxiv.org/pdf/2310.04742v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parameter-efficient-multi-task-model-fusion","repo_url":"https://github.com/tanganke/peta","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"task-arithmetic","task_name":"Task Arithmetic"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.04742","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.04742"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tanganke/peta","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"b3de151e399cc52f","entry":"dict_params_to_tuple","repo":"tanganke/peta","repo_kind":"official","path":"peta/models/LinearizedModel.py","file_url":"https://github.com/tanganke/peta/blob/HEAD/peta/models/LinearizedModel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3de151e399cc52f"}},{"code_sha256_prefix":"8a287ff0d115b9f7","entry":"load_dataset","repo":"tanganke/peta","repo_kind":"official","path":"peta/lorahub/algorithm.py","file_url":"https://github.com/tanganke/peta/blob/HEAD/peta/lorahub/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a287ff0d115b9f7"}},{"code_sha256_prefix":"4fa148bde5c27d5c","entry":"preprocess_function","repo":"tanganke/peta","repo_kind":"official","path":"peta/lorahub/algorithm.py","file_url":"https://github.com/tanganke/peta/blob/HEAD/peta/lorahub/algorithm.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4fa148bde5c27d5c"}},{"code_sha256_prefix":"ebcf48623d2475b7","entry":"load_base_model_and_lora_modules","repo":"tanganke/peta","repo_kind":"official","path":"peta/lorahub/algorithm.py","file_url":"https://github.com/tanganke/peta/blob/HEAD/peta/lorahub/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebcf48623d2475b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}