{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/surgeryv2-bridging-the-gap-between-model","title":"SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery","arxiv_id":"2410.14389","date":"2024-10-18","proceeding":null,"authors":["Enneng Yang","Li Shen","Zhenyi Wang","Guibing Guo","Xingwei Wang","Xiaocun Cao","Jie Zhang","DaCheng Tao"],"abstract":"Model merging-based multitask learning (MTL) offers a promising approach for performing MTL by merging multiple expert models without requiring access to raw training data. However, in this paper, we examine the merged model's representation distribution and uncover a critical issue of \"representation bias\". This bias arises from a significant distribution gap between the representations of the merged and expert models, leading to the suboptimal performance of the merged MTL model. To address this challenge, we first propose a representation surgery solution called Surgery. Surgery is a lightweight, task-specific module that aligns the final layer representations of the merged model with those of the expert models, effectively alleviating bias and improving the merged model's performance. Despite these improvements, a performance gap remains compared to the traditional MTL method. Further analysis reveals that representation bias phenomena exist at each layer of the merged model, and aligning representations only in the last layer is insufficient for fully reducing systemic bias because biases introduced at each layer can accumulate and interact in complex ways. To tackle this, we then propose a more comprehensive solution, deep representation surgery (also called SurgeryV2), which mitigates representation bias across all layers, and thus bridges the performance gap between model merging-based MTL and traditional MTL. Finally, we design an unsupervised optimization objective to optimize both the Surgery and SurgeryV2 modules. Our experimental results show that incorporating these modules into state-of-the-art (SOTA) model merging schemes leads to significant performance gains. Notably, our SurgeryV2 scheme reaches almost the same level as individual expert models or the traditional MTL model. The code is available at \\url{https://github.com/EnnengYang/SurgeryV2}.","url_abs":"https://arxiv.org/abs/2410.14389v1","url_pdf":"https://arxiv.org/pdf/2410.14389v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"surgeryv2-bridging-the-gap-between-model","repo_url":"https://github.com/ennengyang/surgeryv2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.14389","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.14389"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/EnnengYang/SurgeryV2","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ennengyang/surgeryv2","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":3,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"2d5d7a12d8de02b7","entry":"cosine_lr","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d5d7a12d8de02b7"}},{"code_sha256_prefix":"f52df35accf87444","entry":"get_merging_cofficients","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/merging_cofficient.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/merging_cofficient.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f52df35accf87444"}},{"code_sha256_prefix":"2ca594138c45f6d6","entry":"l1losses","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/merging_model.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/merging_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ca594138c45f6d6"}},{"code_sha256_prefix":"8adbe30621fbeb18","entry":"make_functional","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/merging_model.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/merging_model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8adbe30621fbeb18"}},{"code_sha256_prefix":"0c501eba6aaad5fa","entry":"softmax_entropy","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/merging_model.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/merging_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c501eba6aaad5fa"}},{"code_sha256_prefix":"1e9d453dc1605e8c","entry":"split_train_into_train_val","repo":"ennengyang/surgeryv2","repo_kind":"official","path":"src/datasets/registry.py","file_url":"https://github.com/ennengyang/surgeryv2/blob/HEAD/src/datasets/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e9d453dc1605e8c"}},{"code_sha256_prefix":"72391ca34e9a8cfe","entry":"state_dict_to_vector","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/ties_merging_utils.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/ties_merging_utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"72391ca34e9a8cfe"}},{"code_sha256_prefix":"bf6efec91354af72","entry":"vector_to_state_dict","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/ties_merging_utils.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/ties_merging_utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf6efec91354af72"}},{"code_sha256_prefix":"2b9cb736cd2df186","entry":"vector_to_state_dict_unsort","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/ties_merging_utils.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/ties_merging_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b9cb736cd2df186"}},{"code_sha256_prefix":"539d129135c1c4c8","entry":"ModelWrapper_Surgery_V2","repo":"ennengyang/surgeryv2","repo_kind":"official","path":"src/merging_model.py","file_url":"https://github.com/ennengyang/surgeryv2/blob/HEAD/src/merging_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"539d129135c1c4c8"}},{"code_sha256_prefix":"aafcff3ad1ccef6e","entry":"accuracy","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aafcff3ad1ccef6e"}},{"code_sha256_prefix":"7d051a7adefedb6c","entry":"torch_load_old","repo":"EnnengYang/SurgeryV2","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/EnnengYang/SurgeryV2/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d051a7adefedb6c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}