{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/merging-experts-into-one-improving","title":"Merging Experts into One: Improving Computational Efficiency of Mixture of Experts","arxiv_id":"2310.09832","date":"2023-10-15","proceeding":null,"authors":["Shwai He","Run-Ze Fan","Liang Ding","Li Shen","Tianyi Zhou","DaCheng Tao"],"abstract":"Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of growing computational cost. Although a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters (e.g., one expert) for each input, its computation escalates significantly if increasing the number of activated experts, limiting its practical utility. Can we retain the advantages of adding more experts without substantially increasing the computational costs? In this paper, we first demonstrate the superiority of selecting multiple experts and then propose a computation-efficient approach called \\textbf{\\texttt{Merging Experts into One}} (MEO), which reduces the computation cost to that of a single expert. Extensive experiments show that MEO significantly improves computational efficiency, e.g., FLOPS drops from 72.0G of vanilla MoE to 28.6G (MEO). Moreover, we propose a token-level attention block that further enhances the efficiency and performance of token-level MEO, e.g., 83.3\\% (MEO) vs. 82.6\\% (vanilla MoE) average score on the GLUE benchmark. Our code will be released upon acceptance. Code will be released at: \\url{https://github.com/Shwai-He/MEO}.","url_abs":"https://arxiv.org/abs/2310.09832v3","url_pdf":"https://arxiv.org/pdf/2310.09832v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"merging-experts-into-one-improving","repo_url":"https://github.com/shwai-he/meo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.09832","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.09832"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shwai-he/meo","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Shwai-He/MEO","reach":{"status":"ok"}}],"summary":{"ran":12,"ran_honours":1,"ran_draft_wrong":3,"unverified":5},"by_repo_kind":{"official":{"samples":21,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"0766417d0a9cc795","entry":"MEO","repo":"Shwai-He/MEO","repo_kind":"official","path":"transformers/models/layers.py","file_url":"https://github.com/Shwai-He/MEO/blob/HEAD/transformers/models/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0766417d0a9cc795"}},{"code_sha256_prefix":"721cc7eedfa0ef65","entry":"TokenAtt","repo":"Shwai-He/MEO","repo_kind":"official","path":"transformers/models/layers.py","file_url":"https://github.com/Shwai-He/MEO/blob/HEAD/transformers/models/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"721cc7eedfa0ef65"}},{"code_sha256_prefix":"bd49c412803e1b6e","entry":"_gates_to_load","repo":"Shwai-He/MEO","repo_kind":"official","path":"transformers/models/layers.py","file_url":"https://github.com/Shwai-He/MEO/blob/HEAD/transformers/models/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd49c412803e1b6e"}},{"code_sha256_prefix":"a089868d27926865","entry":"_prob_in_top_k","repo":"Shwai-He/MEO","repo_kind":"official","path":"transformers/models/layers.py","file_url":"https://github.com/Shwai-He/MEO/blob/HEAD/transformers/models/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a089868d27926865"}},{"code_sha256_prefix":"743ceb9025a7955d","entry":"booleans_processing","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_tf_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_tf_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"743ceb9025a7955d"}},{"code_sha256_prefix":"5168a172bbb363a5","entry":"convert_pytorch_state_dict_to_flax","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5168a172bbb363a5"}},{"code_sha256_prefix":"abbff004073dc773","entry":"cv_squared","repo":"Shwai-He/MEO","repo_kind":"official","path":"transformers/models/layers.py","file_url":"https://github.com/Shwai-He/MEO/blob/HEAD/transformers/models/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abbff004073dc773"}},{"code_sha256_prefix":"3c1eb64dde6b86af","entry":"dummy_loss","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_tf_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_tf_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c1eb64dde6b86af"}},{"code_sha256_prefix":"26eb4e2856b3d7a1","entry":"generate_batch_splits","repo":"shwai-he/meo","repo_kind":"official","path":"tasks/language-modeling/run_bart_dlm.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/tasks/language-modeling/run_bart_dlm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26eb4e2856b3d7a1"}},{"code_sha256_prefix":"ccaa0edba373622c","entry":"get_imports","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/dynamic_module_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ccaa0edba373622c"}},{"code_sha256_prefix":"dd8f7751ef859581","entry":"infer_metric_tags_from_eval_results","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modelcard.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd8f7751ef859581"}},{"code_sha256_prefix":"6a3959a0ffd36577","entry":"keras_serializable","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_tf_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_tf_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a3959a0ffd36577"}},{"code_sha256_prefix":"626cbdc393d06c43","entry":"noisy_top_k_gating","repo":"Shwai-He/MEO","repo_kind":"official","path":"transformers/models/layers.py","file_url":"https://github.com/Shwai-He/MEO/blob/HEAD/transformers/models/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"626cbdc393d06c43"}},{"code_sha256_prefix":"a4491b45d19b7a39","entry":"parse_keras_history","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modelcard.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4491b45d19b7a39"}},{"code_sha256_prefix":"a66d2b78670c85e1","entry":"quick_gelu","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_flax_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a66d2b78670c85e1"}},{"code_sha256_prefix":"61779b0406a59cfc","entry":"rename_key_and_reshape_tensor","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61779b0406a59cfc"}},{"code_sha256_prefix":"9d98806acb3144b7","entry":"dtype_byte_size","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_flax_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d98806acb3144b7"}},{"code_sha256_prefix":"e3ce70ca9258800e","entry":"flax_shard_checkpoint","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modeling_flax_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3ce70ca9258800e"}},{"code_sha256_prefix":"00cc853a73f3aeb6","entry":"get_relative_import_files","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/dynamic_module_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00cc853a73f3aeb6"}},{"code_sha256_prefix":"450aeffe7f80f917","entry":"get_relative_imports","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/dynamic_module_utils.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"450aeffe7f80f917"}},{"code_sha256_prefix":"ec7b1528a936234d","entry":"is_hf_dataset","repo":"shwai-he/meo","repo_kind":"official","path":"transformers/modelcard.py","file_url":"https://github.com/shwai-he/meo/blob/HEAD/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec7b1528a936234d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}