{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/modeling-multi-task-model-merging-as-adaptive","title":"Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent","arxiv_id":"2501.01230","date":"2025-01-02","proceeding":null,"authors":["Yongxian Wei","Anke Tang","Li Shen","Chun Yuan","Xiaochun Cao"],"abstract":"Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data. Existing methods attempt to alleviate task conflicts by sparsifying task vectors or promoting orthogonality among them. However, they overlook the fundamental requirement of model merging: ensuring the merged model performs comparably to task-specific models on respective tasks. We find these methods inevitably discard task-specific information that, while causing conflicts, is crucial for performance. Based on our findings, we frame model merging as a constrained optimization problem ($\\textit{i.e.}$, minimizing the gap between the merged model and individual models, subject to the constraint of retaining shared knowledge) and solve it via adaptive projective gradient descent. Specifically, we align the merged model with individual models by decomposing and reconstituting the loss function, alleviating conflicts through $\\textit{data-free}$ optimization of task vectors. To retain shared knowledge, we optimize this objective by projecting gradients within a $\\textit{shared subspace}$ spanning all tasks. Moreover, we view merging coefficients as adaptive learning rates and propose a task-aware, training-free strategy. Experiments show that our plug-and-play approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.","url_abs":"https://arxiv.org/abs/2501.01230v2","url_pdf":"https://arxiv.org/pdf/2501.01230v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"modeling-multi-task-model-merging-as-adaptive","repo_url":"https://github.com/walkerworldpeace/doge","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.01230","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.01230"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/walkerworldpeace/doge","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6866d2a6d69a1ecd","entry":"recombine_modeldict","repo":"walkerworldpeace/doge","repo_kind":"official","path":"fusion_bench/method/model_recombination.py","file_url":"https://github.com/walkerworldpeace/doge/blob/HEAD/fusion_bench/method/model_recombination.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6866d2a6d69a1ecd"}},{"code_sha256_prefix":"b5d44b089a2cdd0b","entry":"recombine_modellist","repo":"walkerworldpeace/doge","repo_kind":"official","path":"fusion_bench/method/model_recombination.py","file_url":"https://github.com/walkerworldpeace/doge/blob/HEAD/fusion_bench/method/model_recombination.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5d44b089a2cdd0b"}},{"code_sha256_prefix":"3e6d09646a3092dc","entry":"recombine_state_dict","repo":"walkerworldpeace/doge","repo_kind":"official","path":"fusion_bench/method/model_recombination.py","file_url":"https://github.com/walkerworldpeace/doge/blob/HEAD/fusion_bench/method/model_recombination.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e6d09646a3092dc"}},{"code_sha256_prefix":"7076dab2ff2fb2cd","entry":"get_attr","repo":"walkerworldpeace/doge","repo_kind":"official","path":"fusion_bench/models/wrappers/layer_wise_fusion_doge_ta.py","file_url":"https://github.com/walkerworldpeace/doge/blob/HEAD/fusion_bench/models/wrappers/layer_wise_fusion_doge_ta.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7076dab2ff2fb2cd"}},{"code_sha256_prefix":"694c0fef8290f250","entry":"get_layer_wise_weights","repo":"walkerworldpeace/doge","repo_kind":"official","path":"fusion_bench/models/wrappers/layer_wise_fusion_doge_ta.py","file_url":"https://github.com/walkerworldpeace/doge/blob/HEAD/fusion_bench/models/wrappers/layer_wise_fusion_doge_ta.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"694c0fef8290f250"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}