{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-weight-averaged-model-merging","title":"Rethinking Weight-Averaged Model-merging","arxiv_id":"2411.09263","date":"2024-11-14","proceeding":null,"authors":["Hu Wang","Congbo Ma","Ibrahim Almakky","Ian Reid","Gustavo Carneiro","Mohammad Yaqub"],"abstract":"Model-merging has emerged as a powerful approach in deep learning, capable of enhancing model performance without any training. However, the underlying mechanisms that explain its effectiveness remain largely unexplored. In this paper, we investigate this technique from three novel perspectives to empirically provide deeper insights into why and how weight-averaged model-merging~\\cite{wortsman2022soups} works: (1) we examine the intrinsic patterns captured by the learning of the model weights, and we are the first to connect that these weights encode structured with why weight-averaged model merging can work; (2) we investigate averaging on weights versus averaging on features, providing analyses from the view of diverse architecture comparisons on multiple datasets; and (3) we explore the impact on model-merging prediction stability in terms of changing the parameter magnitude, revealing insights into the way of weight averaging works as regularization by showing the robustness across different parameter scales. The code is available at https://github.com/billhhh/Rethink-Merge.","url_abs":"https://arxiv.org/abs/2411.09263v4","url_pdf":"https://arxiv.org/pdf/2411.09263v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-weight-averaged-model-merging","repo_url":"https://github.com/billhhh/rethink-merge","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.09263","atlas_url":"https://app.syntology.ai/?focus=2411.09263","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.09263"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/billhhh/rethink-merge","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"436fbeb218857c5b","entry":"maybe_dictionarize_batch","repo":"billhhh/rethink-merge","repo_kind":"official","path":"utils.py","file_url":"https://github.com/billhhh/rethink-merge/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"436fbeb218857c5b"}},{"code_sha256_prefix":"94729a43193ff2ff","entry":"get_images_tensor","repo":"billhhh/rethink-merge","repo_kind":"official","path":"Wt_Patterns/Linear_vis_tinyimgnet.py","file_url":"https://github.com/billhhh/rethink-merge/blob/HEAD/Wt_Patterns/Linear_vis_tinyimgnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94729a43193ff2ff"}},{"code_sha256_prefix":"3cc7b5392fdb2336","entry":"get_model_from_sd","repo":"billhhh/rethink-merge","repo_kind":"official","path":"utils.py","file_url":"https://github.com/billhhh/rethink-merge/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3cc7b5392fdb2336"}},{"code_sha256_prefix":"3c7549cdcc9a348f","entry":"get_model_from_sd_org","repo":"billhhh/rethink-merge","repo_kind":"official","path":"utils.py","file_url":"https://github.com/billhhh/rethink-merge/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c7549cdcc9a348f"}},{"code_sha256_prefix":"c0e8f1254618263d","entry":"imshow","repo":"billhhh/rethink-merge","repo_kind":"official","path":"Wt_Patterns/ave_tinyimgnet.py","file_url":"https://github.com/billhhh/rethink-merge/blob/HEAD/Wt_Patterns/ave_tinyimgnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0e8f1254618263d"}},{"code_sha256_prefix":"a56f6e2e74ec1a03","entry":"preprocess","repo":"billhhh/rethink-merge","repo_kind":"official","path":"Wt_Patterns/ave_tinyimgnet.py","file_url":"https://github.com/billhhh/rethink-merge/blob/HEAD/Wt_Patterns/ave_tinyimgnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a56f6e2e74ec1a03"}},{"code_sha256_prefix":"ff3ee610b4ac419c","entry":"validate_model","repo":"billhhh/rethink-merge","repo_kind":"official","path":"Wt_vs_Ft/cifar10_merge.py","file_url":"https://github.com/billhhh/rethink-merge/blob/HEAD/Wt_vs_Ft/cifar10_merge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff3ee610b4ac419c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}