{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/demystifying-the-compression-of-mixture-of","title":"Demystifying the Compression of Mixture-of-Experts Through a Unified Framework","arxiv_id":"2406.02500","date":"2024-06-04","proceeding":null,"authors":["Shwai He","Daize Dong","Liang Ding","Ang Li"],"abstract":"Scaling large language models has revolutionized the performance across diverse domains, yet the continual growth in model size poses significant challenges for real-world deployment. The Mixture of Experts (MoE) approach addresses this by dynamically selecting and activating only a subset of experts, significantly reducing computational costs while maintaining high performance. However, MoE introduces potential redundancy (e.g., parameters) and extra costs (e.g., communication overhead). Despite numerous compression techniques developed for mitigating the redundancy in dense models, the compression of MoE remains under-explored. We first bridge this gap with a cutting-edge unified framework that not only seamlessly integrates mainstream compression methods but also helps systematically understand MoE compression. This framework approaches compression from two perspectives: Expert Slimming which compresses individual experts and Expert Trimming which removes structured modules. Within this framework, we explore the optimization space unexplored by existing methods,and further introduce aggressive Expert Trimming techniques, i.e., Layer Drop and Block Drop, to eliminate redundancy at larger scales. Based on these insights,we present a comprehensive recipe to guide practitioners in compressing MoE effectively. Extensive experimental results demonstrate the effectiveness of the compression methods under our framework and the proposed recipe, achieving a 6.05x speedup and only 20.0GB memory usage while maintaining over 92% of performance on Mixtral-8x7B. Code is released at \\url{https://github.com/DaizeDong/Unified-MoE-Compression}.","url_abs":"https://arxiv.org/abs/2406.02500v2","url_pdf":"https://arxiv.org/pdf/2406.02500v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"demystifying-the-compression-of-mixture-of","repo_url":"https://github.com/daizedong/unified-moe-compression","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.02500","atlas_url":"https://app.syntology.ai/?focus=2406.02500","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02500"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/daizedong/unified-moe-compression","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7b483277d903ceb1","entry":"check_sparsity_from_state_dict","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/utils.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b483277d903ceb1"}},{"code_sha256_prefix":"f7e6b2633db22c17","entry":"fill_missing_values_for_non_moe_layers","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/expert_drop.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/expert_drop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f7e6b2633db22c17"}},{"code_sha256_prefix":"6031b360d9fd7f04","entry":"find_modules","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/utils.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6031b360d9fd7f04"}},{"code_sha256_prefix":"d857212c7c79f1fe","entry":"find_moe_expert_linears","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/utils.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d857212c7c79f1fe"}},{"code_sha256_prefix":"8db0c1b19aba233f","entry":"load_json","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/io.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8db0c1b19aba233f"}},{"code_sha256_prefix":"858343dca0bec6c6","entry":"apply_moe_weights","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"AutoAWQ/awq/modules/fused/moe.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/AutoAWQ/awq/modules/fused/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"858343dca0bec6c6"}},{"code_sha256_prefix":"a1324de2a7fd6559","entry":"fused_topk","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"AutoAWQ/awq/modules/fused/moe.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/AutoAWQ/awq/modules/fused/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1324de2a7fd6559"}},{"code_sha256_prefix":"68b3f990b0ec61d6","entry":"moe_align_block_size","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"AutoAWQ/awq/modules/fused/moe.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/AutoAWQ/awq/modules/fused/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68b3f990b0ec61d6"}},{"code_sha256_prefix":"bdface3275751bca","entry":"prune_magnitude","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/prune.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdface3275751bca"}},{"code_sha256_prefix":"3c36697395392b37","entry":"prune_sparsegpt","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/prune.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c36697395392b37"}},{"code_sha256_prefix":"477c9e48c3c1bac2","entry":"prune_wanda","repo":"daizedong/unified-moe-compression","repo_kind":"official","path":"src/llmtuner/compression/prune/prune.py","file_url":"https://github.com/daizedong/unified-moe-compression/blob/HEAD/src/llmtuner/compression/prune/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"477c9e48c3c1bac2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}