{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/localizing-task-information-for-improved","title":"Localizing Task Information for Improved Model Merging and Compression","arxiv_id":"2405.07813","date":"2024-05-13","proceeding":null,"authors":["Ke Wang","Nikolaos Dimitriadis","Guillermo Ortiz-Jimenez","François Fleuret","Pascal Frossard"],"abstract":"Model merging and task arithmetic have emerged as promising scalable approaches to merge multiple single-task checkpoints to one multi-task model, but their applicability is reduced by significant performance loss. Previous works have linked these drops to interference in the weight space and erasure of important task-specific features. Instead, in this work we show that the information required to solve each task is still preserved after merging as different tasks mostly use non-overlapping sets of weights. We propose TALL-masks, a method to identify these task supports given a collection of task vectors and show that one can retrieve >99% of the single task accuracy by applying our masks to the multi-task vector, effectively compressing the individual checkpoints. We study the statistics of intersections among constructed masks and reveal the existence of selfish and catastrophic weights, i.e., parameters that are important exclusively to one task and irrelevant to all tasks but detrimental to multi-task fusion. For this reason, we propose Consensus Merging, an algorithm that eliminates such weights and improves the general performance of existing model merging approaches. Our experiments in vision and NLP benchmarks with up to 20 tasks, show that Consensus Merging consistently improves existing approaches. Furthermore, our proposed compression scheme reduces storage from 57Gb to 8.2Gb while retaining 99.7% of original performance.","url_abs":"https://arxiv.org/abs/2405.07813v1","url_pdf":"https://arxiv.org/pdf/2405.07813v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"localizing-task-information-for-improved","repo_url":"https://github.com/nik-dim/tall_masks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"localizing-task-information-for-improved","repo_url":"https://github.com/duguodong7/model-evolution","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"task-arithmetic","task_name":"Task Arithmetic"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.07813","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.07813"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nik-dim/tall_masks","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/duguodong7/model-evolution","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6d77ee840479c381","entry":"convert","repo":"nik-dim/tall_masks","repo_kind":"official","path":"src/datasets/cifar10.py","file_url":"https://github.com/nik-dim/tall_masks/blob/HEAD/src/datasets/cifar10.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d77ee840479c381"}},{"code_sha256_prefix":"b1e9e0d3d7d7e615","entry":"maybe_dictionarize","repo":"nik-dim/tall_masks","repo_kind":"official","path":"src/datasets/common.py","file_url":"https://github.com/nik-dim/tall_masks/blob/HEAD/src/datasets/common.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1e9e0d3d7d7e615"}},{"code_sha256_prefix":"241a2a00ba0c307b","entry":"pretify_classname","repo":"nik-dim/tall_masks","repo_kind":"official","path":"src/datasets/eurosat.py","file_url":"https://github.com/nik-dim/tall_masks/blob/HEAD/src/datasets/eurosat.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"241a2a00ba0c307b"}},{"code_sha256_prefix":"3b97c12751c0cf5b","entry":"get_features","repo":"nik-dim/tall_masks","repo_kind":"official","path":"src/datasets/common.py","file_url":"https://github.com/nik-dim/tall_masks/blob/HEAD/src/datasets/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b97c12751c0cf5b"}},{"code_sha256_prefix":"238ff32fcbf410d6","entry":"get_features_helper","repo":"nik-dim/tall_masks","repo_kind":"official","path":"src/datasets/common.py","file_url":"https://github.com/nik-dim/tall_masks/blob/HEAD/src/datasets/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"238ff32fcbf410d6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}