{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/realistic-evaluation-of-model-merging-for","title":"Realistic Evaluation of Model Merging for Compositional Generalization","arxiv_id":"2409.18314","date":"2024-09-26","proceeding":null,"authors":["Derek Tam","Yash Kant","Brian Lester","Igor Gilitschenski","Colin Raffel"],"abstract":"Merging has become a widespread way to cheaply combine individual models into a single model that inherits their capabilities and attains better performance. This popularity has spurred rapid development of many new merging methods, which are typically validated in disparate experimental settings and frequently differ in the assumptions made about model architecture, data availability, and computational budget. In this work, we characterize the relative merits of different merging methods by evaluating them in a shared experimental setting and precisely identifying the practical requirements of each method. Specifically, our setting focuses on using merging for compositional generalization of capabilities in image classification, image generation, and natural language processing. Additionally, we measure the computational costs of different merging methods as well as how they perform when scaling the number of models being merged. Taken together, our results clarify the state of the field of model merging and provide a comprehensive and rigorous experimental setup to test new methods.","url_abs":"https://arxiv.org/abs/2409.18314v1","url_pdf":"https://arxiv.org/pdf/2409.18314v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"realistic-evaluation-of-model-merging-for","repo_url":"https://github.com/r-three/realistic_evaluation_of_model_merging_for_compositional_generalization","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.18314","atlas_url":"https://app.syntology.ai/?focus=2409.18314","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.18314"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/r-three/realistic_evaluation_of_model_merging_for_compositional_generalization","reach":null}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9fbff4f411c485ac","entry":"get_statisticsDir","repo":"r-three/realistic_evaluation_of_model_merging_for_compositional_generalization","repo_kind":"official","path":"src/merging/save_statistic.py","file_url":"https://github.com/r-three/realistic_evaluation_of_model_merging_for_compositional_generalization/blob/HEAD/src/merging/save_statistic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fbff4f411c485ac"}},{"code_sha256_prefix":"225c26ebc567caaa","entry":"topk_values_mask","repo":"r-three/realistic_evaluation_of_model_merging_for_compositional_generalization","repo_kind":"official","path":"src/merging/save_statistic.py","file_url":"https://github.com/r-three/realistic_evaluation_of_model_merging_for_compositional_generalization/blob/HEAD/src/merging/save_statistic.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"225c26ebc567caaa"}},{"code_sha256_prefix":"db20c6a9f6d0a3cd","entry":"compute_taskVectors","repo":"r-three/realistic_evaluation_of_model_merging_for_compositional_generalization","repo_kind":"official","path":"src/merging/save_statistic.py","file_url":"https://github.com/r-three/realistic_evaluation_of_model_merging_for_compositional_generalization/blob/HEAD/src/merging/save_statistic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db20c6a9f6d0a3cd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}