{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-free-pretrained-model-merging","title":"Training-Free Pretrained Model Merging","arxiv_id":"2403.01753","date":"2024-03-04","proceeding":"CVPR 2024 1","authors":["Zhengqi Xu","Ke Yuan","Huiqiong Wang","Yong Wang","Mingli Song","Jie Song"],"abstract":"Recently, model merging techniques have surfaced as a solution to combine multiple single-talent models into a single multi-talent model. However, previous endeavors in this field have either necessitated additional training or fine-tuning processes, or require that the models possess the same pre-trained initialization. In this work, we identify a common drawback in prior works w.r.t. the inconsistency of unit similarity in the weight space and the activation space. To address this inconsistency, we propose an innovative model merging framework, coined as merging under dual-space constraints (MuDSC). Specifically, instead of solely maximizing the objective of a single space, we advocate for the exploration of permutation matrices situated in a region with a unified high similarity in the dual space, achieved through the linear combination of activation and weight similarity matrices. In order to enhance usability, we have also incorporated adaptations for group structure, including Multi-Head Attention and Group Normalization. Comprehensive experimental comparisons demonstrate that MuDSC can significantly boost the performance of merged models with various task combinations and architectures. Furthermore, the visualization of the merged model within the multi-task loss landscape reveals that MuDSC enables the merged model to reside in the overlapping segment, featuring a unified lower loss for each task. Our code is publicly available at https://github.com/zju-vipa/training_free_model_merging.","url_abs":"https://arxiv.org/abs/2403.01753v3","url_pdf":"https://arxiv.org/pdf/2403.01753v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-free-pretrained-model-merging","repo_url":"https://github.com/zju-vipa/training_free_model_merging","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"group-normalization","method_name":"Group Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.01753","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.01753"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zju-vipa/training_free_model_merging","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"ee4e6678eafd3832","entry":"WeightFusion","repo":"zju-vipa/training_free_model_merging","repo_kind":"official","path":"heterogeneous_tasks/mudsc.py","file_url":"https://github.com/zju-vipa/training_free_model_merging/blob/HEAD/heterogeneous_tasks/mudsc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee4e6678eafd3832"}},{"code_sha256_prefix":"b7848359c35ca717","entry":"refine_perm","repo":"zju-vipa/training_free_model_merging","repo_kind":"official","path":"heterogeneous_tasks/mudsc.py","file_url":"https://github.com/zju-vipa/training_free_model_merging/blob/HEAD/heterogeneous_tasks/mudsc.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7848359c35ca717"}},{"code_sha256_prefix":"ac28dcec5c8aa992","entry":"remove_col","repo":"zju-vipa/training_free_model_merging","repo_kind":"official","path":"heterogeneous_tasks/mudsc.py","file_url":"https://github.com/zju-vipa/training_free_model_merging/blob/HEAD/heterogeneous_tasks/mudsc.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac28dcec5c8aa992"}},{"code_sha256_prefix":"bf4f3e9d597b9f91","entry":"CovarianceMetric","repo":"zju-vipa/training_free_model_merging","repo_kind":"official","path":"heterogeneous_tasks/mudsc.py","file_url":"https://github.com/zju-vipa/training_free_model_merging/blob/HEAD/heterogeneous_tasks/mudsc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf4f3e9d597b9f91"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}