{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-glue-democratized-llm-scaling-for-a","title":"Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild","arxiv_id":"2410.05357","date":"2024-10-07","proceeding":null,"authors":["Xinyu Zhao","Guoheng Sun","Ruisi Cai","Yukun Zhou","Pingzhi Li","Peihao Wang","Bowen Tan","Yexiao He","Li Chen","Yi Liang","Beidi Chen","Binhang Yuan","Hongyi Wang","Ang Li","Zhangyang Wang","Tianlong Chen"],"abstract":"As Large Language Models (LLMs) excel across tasks and specialized domains, scaling LLMs based on existing models has garnered significant attention, which faces the challenge of decreasing performance when combining disparate models. Various techniques have been proposed for the aggregation of pre-trained LLMs, including model merging, Mixture-of-Experts, and stacking. Despite their merits, a comprehensive comparison and synergistic application of them to a diverse model zoo is yet to be adequately addressed. In light of this research gap, this paper introduces Model-GLUE, a holistic LLM scaling guideline. First, our work starts with a benchmarking of existing LLM scaling techniques, especially selective merging, and variants of mixture. Utilizing the insights from the benchmark results, we formulate an optimal strategy for the selection and aggregation of a heterogeneous model zoo characterizing different architectures and initialization.Our methodology involves the clustering of mergeable models and optimal merging strategy selection, and the integration of clusters through a model mixture. Finally, evidenced by our experiments on a diverse Llama-2-based model zoo, Model-GLUE shows an average performance enhancement of 5.61%, achieved without additional training. Codes are available at: https://github.com/Model-GLUE/Model-GLUE.","url_abs":"https://arxiv.org/abs/2410.05357v2","url_pdf":"https://arxiv.org/pdf/2410.05357v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-glue-democratized-llm-scaling-for-a","repo_url":"https://github.com/model-glue/model-glue","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.05357","atlas_url":"https://app.syntology.ai/?focus=2410.05357","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05357"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Model-GLUE/Model-GLUE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/model-glue/model-glue","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran":6,"ran_draft_wrong":2,"unverified":6},"by_repo_kind":{"official":{"samples":16,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"Model-GLUE/Model-GLUE","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/Model-GLUE/Model-GLUE/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"5f809614ad45d87e","entry":"MixtralModelLevelHybridConfig","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f809614ad45d87e"}},{"code_sha256_prefix":"168afe1c35dcbf7a","entry":"MixtralModelLevelHybridMLP","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"168afe1c35dcbf7a"}},{"code_sha256_prefix":"61211405adf0232e","entry":"MixtralModelLevelHybridRMSNorm","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61211405adf0232e"}},{"code_sha256_prefix":"2af3f93fb1f05187","entry":"MixtralModelLevelHybridRotaryEmbedding","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2af3f93fb1f05187"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"Model-GLUE/Model-GLUE","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/Model-GLUE/Model-GLUE/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"20a7cc804eb22661","entry":"hash_args","repo":"Model-GLUE/Model-GLUE","repo_kind":"official","path":"lm_eval/api/model.py","file_url":"https://github.com/Model-GLUE/Model-GLUE/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a7cc804eb22661"}},{"code_sha256_prefix":"29ee1d8c120f674a","entry":"remove_after_return","repo":"Model-GLUE/Model-GLUE","repo_kind":"official","path":"bigcode_eval/utils.py","file_url":"https://github.com/Model-GLUE/Model-GLUE/blob/HEAD/bigcode_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29ee1d8c120f674a"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"Model-GLUE/Model-GLUE","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/Model-GLUE/Model-GLUE/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"ac86a6f136bea277","entry":"update_code_gens","repo":"Model-GLUE/Model-GLUE","repo_kind":"official","path":"bigcode_eval/utils.py","file_url":"https://github.com/Model-GLUE/Model-GLUE/blob/HEAD/bigcode_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac86a6f136bea277"}},{"code_sha256_prefix":"b7a56ddb57bcb75d","entry":"MixtralModelLevelHybridAttention","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7a56ddb57bcb75d"}},{"code_sha256_prefix":"10f90dfc6066b0b8","entry":"MixtralModelLevelHybridDenseDecoderLayer","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10f90dfc6066b0b8"}},{"code_sha256_prefix":"c81e094f251fdc6a","entry":"MixtralModelLevelHybridFlashAttention2","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c81e094f251fdc6a"}},{"code_sha256_prefix":"2315328e9bcd8f59","entry":"MixtralModelLevelHybridSdpaAttention","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2315328e9bcd8f59"}},{"code_sha256_prefix":"3cc021576be7867c","entry":"MixtralModelLevelHybridSparseDecoderLayer","repo":"model-glue/model-glue","repo_kind":"official","path":"mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","file_url":"https://github.com/model-glue/model-glue/blob/HEAD/mixture/mixtral_model_level/modeling_mixtral_model_level_hybrid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cc021576be7867c"}},{"code_sha256_prefix":"59f94e66fd20d85d","entry":"complete_code","repo":"Model-GLUE/Model-GLUE","repo_kind":"official","path":"bigcode_eval/utils.py","file_url":"https://github.com/Model-GLUE/Model-GLUE/blob/HEAD/bigcode_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59f94e66fd20d85d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}