{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/merge-then-compress-demystify-efficient-smoe","title":"Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy","arxiv_id":"2310.01334","date":"2023-10-02","proceeding":null,"authors":["Pingzhi Li","Zhenyu Zhang","Prateek Yadav","Yi-Lin Sung","Yu Cheng","Mohit Bansal","Tianlong Chen"],"abstract":"Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse. Therefore, vanilla SMoE models are memory inefficient and non-scalable, especially for resource-constrained downstream scenarios. In this paper, we ask: Can we craft a compact SMoE model by consolidating expert information? What is the best recipe to merge multiple experts into fewer but more knowledgeable experts? Our pilot investigation reveals that conventional model merging methods fail to be effective in such expert merging for SMoE. The potential reasons are: (1) redundant information overshadows critical experts; (2) appropriate neuron permutation for each expert is missing to bring all of them in alignment. To address this, we propose M-SMoE, which leverages routing statistics to guide expert merging. Specifically, it starts with neuron permutation alignment for experts; then, dominant experts and their \"group members\" are formed; lastly, every expert group is merged into a single expert by utilizing each expert's activation frequency as their weight for merging, thus diminishing the impact of insignificant experts. Moreover, we observed that our proposed merging promotes a low dimensionality in the merged expert's weight space, naturally paving the way for additional compression. Hence, our final method, MC-SMoE (i.e., Merge, then Compress SMoE), further decomposes the merged experts into low-rank and structural sparse alternatives. Extensive experiments across 8 benchmarks validate the effectiveness of MC-SMoE. For instance, our MC-SMoE achieves up to 80% memory and a 20% FLOPs reduction, with virtually no loss in performance.","url_abs":"https://arxiv.org/abs/2310.01334v2","url_pdf":"https://arxiv.org/pdf/2310.01334v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"merge-then-compress-demystify-efficient-smoe","repo_url":"https://github.com/unites-lab/mc-smoe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.01334","atlas_url":"https://app.syntology.ai/?focus=2310.01334","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.01334"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/UNITES-Lab/MC-SMoE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/unites-lab/mc-smoe","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_honours":1,"ran_violates":2},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fd729d8d235a3757","entry":"ExpertsGrouperForSwitch","repo":"unites-lab/mc-smoe","repo_kind":"official","path":"mcsmoe/merging/grouping.py","file_url":"https://github.com/unites-lab/mc-smoe/blob/HEAD/mcsmoe/merging/grouping.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd729d8d235a3757"}},{"code_sha256_prefix":"e5839c1e066ffc90","entry":"generate_random_group_labels","repo":"unites-lab/mc-smoe","repo_kind":"official","path":"mcsmoe/merging/grouping.py","file_url":"https://github.com/unites-lab/mc-smoe/blob/HEAD/mcsmoe/merging/grouping.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5839c1e066ffc90"}},{"code_sha256_prefix":"654f3d919093632b","entry":"sanitize_merging_layers","repo":"unites-lab/mc-smoe","repo_kind":"official","path":"mcsmoe/msmoe-merging.py","file_url":"https://github.com/unites-lab/mc-smoe/blob/HEAD/mcsmoe/msmoe-merging.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"654f3d919093632b"}},{"code_sha256_prefix":"5a9a3df7db5c12d8","entry":"sanitize_merging_layers","repo":"unites-lab/mc-smoe","repo_kind":"official","path":"mcsmoe/losparse-downstream.py","file_url":"https://github.com/unites-lab/mc-smoe/blob/HEAD/mcsmoe/losparse-downstream.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a9a3df7db5c12d8"}},{"code_sha256_prefix":"97f21e77592be7a0","entry":"sanitize_merging_layers","repo":"UNITES-Lab/MC-SMoE","repo_kind":"official","path":"mcsmoe/prune-non-core-experts-and-kd-downstream.py","file_url":"https://github.com/UNITES-Lab/MC-SMoE/blob/HEAD/mcsmoe/prune-non-core-experts-and-kd-downstream.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97f21e77592be7a0"}},{"code_sha256_prefix":"cd6cf7dbefcd9099","entry":"sanitize_pruning_layers","repo":"UNITES-Lab/MC-SMoE","repo_kind":"official","path":"mcsmoe/switch-finetune-with-task-specific-pruning.py","file_url":"https://github.com/UNITES-Lab/MC-SMoE/blob/HEAD/mcsmoe/switch-finetune-with-task-specific-pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd6cf7dbefcd9099"}},{"code_sha256_prefix":"e4c849c6d01182d2","entry":"sanitize_pruning_layers","repo":"UNITES-Lab/MC-SMoE","repo_kind":"official","path":"mcsmoe/fsgpt-zero-shot-losparse.py","file_url":"https://github.com/UNITES-Lab/MC-SMoE/blob/HEAD/mcsmoe/fsgpt-zero-shot-losparse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4c849c6d01182d2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}