{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-18627","title":"PACT: Preserving Anchored Cores in Task-vectors for Model Merging","arxiv_id":"2606.18627","date":"2026-06-17","proceeding":null,"authors":["Ningyuan Shi","Zhipeng Zhou","Hao Wang","Chunyan Miao","Peilin Zhao"],"abstract":"Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model. Most existing model merging approaches follow the Task Arithmetic paradigm, which decomposes fine-tuned weights into pre-trained parameters and task vectors, and performs merging exclusively in the task-vector space. The effectiveness of this paradigm implicitly relies on the assumption that task-specific knowledge is encoded solely within task vectors. We argue that this assumption generally does not hold due to the intrinsic task preferences of pre-trained models. Specifically, we identify \\textbf{Load-Bearing Wall (LBW) dimensions}, namely some task-critical knowledge that remains embedded in the pre-trained weights rather than being fully transferred into task vectors. We characterize LBW dimensions from both scalar-weight and subspace perspectives, thereby covering the major paradigms of existing model merging methods. Our analysis reveals that, by ignoring LBW dimensions, task-vector-based approaches fail to fully resolve task conflicts and may inadvertently damage task-specific knowledge encoded in the pre-trained model, leading to degradation. To address this issue, we propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their orthogonal complements with the subspace of the pre-trained weights. These aligned subspace components are then removed from the task vectors before applying existing model merging algorithms. Furthermore, we develop an efficient variant based on randomized SVD to improve scalability. PACT can be seamlessly integrated with existing methods. Extensive experiments across multiple benchmarks demonstrate that PACT consistently enhances mainstream model merging approaches and establishes new state-of-the-art performance.","url_abs":"https://arxiv.org/abs/2606.18627","url_pdf":"https://arxiv.org/pdf/2606.18627","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.18627","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.18627"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/dajvmasonjason-glitch/pact","reach":null}],"summary":{"ran_draft_wrong":7,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"f5a10554cb6c878d","entry":"compute_implicit_reliance_space","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5a10554cb6c878d"}},{"code_sha256_prefix":"b4fe04bf747c4c0a","entry":"extract_pretrained_core_space","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4fe04bf747c4c0a"}},{"code_sha256_prefix":"2c5476ffe4253d95","entry":"extract_pretrained_core_space_fixed","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c5476ffe4253d95"}},{"code_sha256_prefix":"c45e72e6a0437cd1","entry":"extract_task_explicit_space","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c45e72e6a0437cd1"}},{"code_sha256_prefix":"0b2d797470b869ac","entry":"extract_task_explicit_space_fixed","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b2d797470b869ac"}},{"code_sha256_prefix":"7e6f89c36673d97d","entry":"orthogonal_core_filtering","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e6f89c36673d97d"}},{"code_sha256_prefix":"4bc49ed9a5df5ff1","entry":"select_pact_layer_keys","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bc49ed9a5df5ff1"}},{"code_sha256_prefix":"dac34dc95f0392af","entry":"pact_ta_merge","repo":"dajvmasonjason-glitch/pact","repo_kind":"found_in_text","path":"full-finetune/src/utils/pact_utils.py","file_url":"https://github.com/dajvmasonjason-glitch/pact/blob/HEAD/full-finetune/src/utils/pact_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dac34dc95f0392af"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}