{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethink-model-re-basin-and-the-linear-mode","title":"Vanishing Feature: Diagnosing Model Merging and Beyond","arxiv_id":"2402.05966","date":"2024-02-05","proceeding":null,"authors":["Xingyu Qu","Samuel Horvath"],"abstract":"Model merging offers an efficient way to combine pre-trained neural networks but often suffers from inconsistent performance, especially when merging models with different initializations. We identify the ``vanishing feature'' phenomenon, where input-induced features diminish during propagation through the merged model, degrading performance. Through theoretical and empirical analysis, we reveal that this phenomenon underpins challenges like variance collapse and explains techniques like permutation-based merging, post-merging normalization, etc. We show that existing normalization strategies can be enhanced by precisely targeting the vanishing feature issue. Leveraging these insights, we propose the ``Preserve-First Merging'' (PFM) strategy, which focuses on preserving early-layer features, enabling the merged models, for the first time, to outperform the original models in advanced settings without post-training. Furthermore, we demonstrate that the vanishing feature phenomenon extends to other contexts, such as model pruning. Applying post-pruning normalization to mitigate the issue significantly improves one-shot pruning performance at high sparsity, offering a simple and effective post-pruning solution. The code is available at https://github.com/XingyuQu/VF.","url_abs":"https://arxiv.org/abs/2402.05966v3","url_pdf":"https://arxiv.org/pdf/2402.05966v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethink-model-re-basin-and-the-linear-mode","repo_url":"https://github.com/xingyuqu/rethink-re-basin","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"rethink-model-re-basin-and-the-linear-mode","repo_url":"https://github.com/xingyuqu/vf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"linear-mode-connectivity","task_name":"Linear Mode Connectivity"},{"task_slug":"re-basin","task_name":"Re-basin"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.05966","atlas_url":"https://app.syntology.ai/?focus=2402.05966","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05966"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xingyuqu/vf","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xingyuqu/rethink-re-basin","reach":null}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"5a9f46b9b120ef6e","entry":"validate","repo":"xingyuqu/vf","repo_kind":"official","path":"PFM/cifar_cca_experiments.py","file_url":"https://github.com/xingyuqu/vf/blob/HEAD/PFM/cifar_cca_experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a9f46b9b120ef6e"}},{"code_sha256_prefix":"b015dad7216d04cf","entry":"validate_ensemble","repo":"xingyuqu/vf","repo_kind":"official","path":"PFM/cifar_cca_experiments.py","file_url":"https://github.com/xingyuqu/vf/blob/HEAD/PFM/cifar_cca_experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b015dad7216d04cf"}},{"code_sha256_prefix":"258a4b4f429690a1","entry":"fim_diag","repo":"xingyuqu/rethink-re-basin","repo_kind":"official","path":"source/utils/fim.py","file_url":"https://github.com/xingyuqu/rethink-re-basin/blob/HEAD/source/utils/fim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"258a4b4f429690a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}