{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/concrete-subspace-learning-based-interference","title":"Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion","arxiv_id":"2312.06173","date":"2023-12-11","proceeding":null,"authors":["Anke Tang","Li Shen","Yong Luo","Liang Ding","Han Hu","Bo Du","DaCheng Tao"],"abstract":"Merging models fine-tuned from a common, extensively pre-trained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multi-task model that performs well across diverse tasks. Recent research, exemplified by task arithmetic, highlights that this multi-task model can be derived through arithmetic operations on task vectors. Nevertheless, current merging techniques frequently resolve potential conflicts among parameters from task-specific models by evaluating individual attributes, such as the parameters' magnitude or sign, overlooking their collective impact on the overall functionality of the model. In this work, we propose the CONtinuous relaxation of disCRETE (Concrete) subspace learning method to identify a common low-dimensional subspace and utilize its shared information to track the interference problem without sacrificing much performance. Specifically, we model the problem as a bi-level optimization problem and introduce a meta-learning framework to find the Concrete subspace mask through gradient-based techniques. At the upper level, we focus on learning a shared Concrete mask to identify the subspace, while at the inner level, model merging is performed to maximize the performance of the merged model. We conduct extensive experiments on both vision domain and language domain, and the results demonstrate the effectiveness of our method. The code is available at https://github.com/tanganke/subspace_fusion","url_abs":"https://arxiv.org/abs/2312.06173v1","url_pdf":"https://arxiv.org/pdf/2312.06173v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"concrete-subspace-learning-based-interference","repo_url":"https://github.com/tanganke/subspace_fusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"task-arithmetic","task_name":"Task Arithmetic"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.06173","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.06173"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tanganke/subspace_fusion","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"31c0be98853385be","entry":"get_block_wise_weights_by_block_size","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/block_wise_fusion.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/block_wise_fusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31c0be98853385be"}},{"code_sha256_prefix":"835005bb4d36b7aa","entry":"get_block_wise_weights_by_partitions","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/block_wise_fusion.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/block_wise_fusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"835005bb4d36b7aa"}},{"code_sha256_prefix":"65c48146354a3100","entry":"get_layer_wise_weights","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/layer_wise_fusion.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/layer_wise_fusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65c48146354a3100"}},{"code_sha256_prefix":"95fa453992ae3b46","entry":"get_task_wise_weights","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/task_wise_fusion.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/task_wise_fusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95fa453992ae3b46"}},{"code_sha256_prefix":"4f0159d8d299aec4","entry":"gumbel_softmax_sample","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/gumbel.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/gumbel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f0159d8d299aec4"}},{"code_sha256_prefix":"64e4ae217b73cdfa","entry":"make_functional","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/adamerging.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/adamerging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64e4ae217b73cdfa"}},{"code_sha256_prefix":"00e884bd2fc5d361","entry":"sample_gumbel","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/gumbel.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/gumbel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00e884bd2fc5d361"}},{"code_sha256_prefix":"d3c4cec689591885","entry":"softmax_entropy","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/adamerging.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/adamerging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3c4cec689591885"}},{"code_sha256_prefix":"4862418d811e8034","entry":"gumbel_softmax","repo":"tanganke/subspace_fusion","repo_kind":"official","path":"src/gumbel.py","file_url":"https://github.com/tanganke/subspace_fusion/blob/HEAD/src/gumbel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4862418d811e8034"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}