{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deepseekmoe-towards-ultimate-expert","title":"DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models","arxiv_id":"2401.06066","date":"2024-01-11","proceeding":null,"authors":["Damai Dai","Chengqi Deng","Chenggang Zhao","R. X. Xu","Huazuo Gao","Deli Chen","Jiashi Li","Wangding Zeng","Xingkai Yu","Y. Wu","Zhenda Xie","Y. K. Li","Panpan Huang","Fuli Luo","Chong Ruan","Zhifang Sui","Wenfeng Liang"],"abstract":"In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.","url_abs":"https://arxiv.org/abs/2401.06066v1","url_pdf":"https://arxiv.org/pdf/2401.06066v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deepseekmoe-towards-ultimate-expert","repo_url":"https://github.com/deepseek-ai/deepseek-moe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"deepseekmoe-towards-ultimate-expert","repo_url":"https://github.com/kamanphoebe/look-into-moes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"gshard","method_name":"GShard"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.06066","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.06066"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kamanphoebe/look-into-moes","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepseek-ai/deepseek-moe","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/huggingface/tokenizers","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa0e5786a11892c1","entry":"AddAuxiliaryLoss","repo":"kamanphoebe/look-into-moes","repo_kind":"listed","path":"deepseekmoe/modeling_deepseek.py","file_url":"https://github.com/kamanphoebe/look-into-moes/blob/HEAD/deepseekmoe/modeling_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa0e5786a11892c1"}},{"code_sha256_prefix":"e891924b51d92aff","entry":"DeepseekMLP","repo":"kamanphoebe/look-into-moes","repo_kind":"listed","path":"deepseekmoe/modeling_deepseek.py","file_url":"https://github.com/kamanphoebe/look-into-moes/blob/HEAD/deepseekmoe/modeling_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e891924b51d92aff"}},{"code_sha256_prefix":"392ad184ccb1c933","entry":"MoEGate","repo":"kamanphoebe/look-into-moes","repo_kind":"listed","path":"deepseekmoe/modeling_deepseek.py","file_url":"https://github.com/kamanphoebe/look-into-moes/blob/HEAD/deepseekmoe/modeling_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"392ad184ccb1c933"}},{"code_sha256_prefix":"765ee58c7d78fb1b","entry":"build_instruction_prompt","repo":"deepseek-ai/deepseek-moe","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/deepseek-ai/deepseek-moe/blob/HEAD/finetune/finetune.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"765ee58c7d78fb1b"}},{"code_sha256_prefix":"436288f52a3803bb","entry":"get_last_checkpoint","repo":"deepseek-ai/deepseek-moe","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/deepseek-ai/deepseek-moe/blob/HEAD/finetune/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"436288f52a3803bb"}},{"code_sha256_prefix":"9c25f2d50056db80","entry":"preprocess","repo":"deepseek-ai/deepseek-moe","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/deepseek-ai/deepseek-moe/blob/HEAD/finetune/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c25f2d50056db80"}},{"code_sha256_prefix":"163c5909c3ae9d0f","entry":"DeepseekMoE","repo":"kamanphoebe/look-into-moes","repo_kind":"listed","path":"deepseekmoe/modeling_deepseek.py","file_url":"https://github.com/kamanphoebe/look-into-moes/blob/HEAD/deepseekmoe/modeling_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"163c5909c3ae9d0f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}