{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/skywork-moe-a-deep-dive-into-training","title":"Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models","arxiv_id":"2406.06563","date":"2024-06-03","proceeding":null,"authors":["Tianwen Wei","Bo Zhu","Liang Zhao","Cheng Cheng","Biye Li","Weiwei Lü","Peng Cheng","Jianhao Zhang","XiaoYu Zhang","Liang Zeng","Xiaokun Wang","Yutuan Ma","Rui Hu","Shuicheng Yan","Han Fang","Yahui Zhou"],"abstract":"In this technical report, we introduce the training methodologies implemented in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE) large language model (LLM) with 146 billion parameters and 16 experts. It is initialized from the pre-existing dense checkpoints of our Skywork-13B model. We explore the comparative effectiveness of upcycling versus training from scratch initializations. Our findings suggest that the choice between these two approaches should consider both the performance of the existing dense checkpoints and the MoE training budget. We highlight two innovative techniques: gating logit normalization, which improves expert diversification, and adaptive auxiliary loss coefficients, allowing for layer-specific adjustment of auxiliary loss coefficients. Our experimental results validate the effectiveness of these methods. Leveraging these techniques and insights, we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile corpus. The evaluation results demonstrate that our model delivers strong performance across a wide range of benchmarks.","url_abs":"https://arxiv.org/abs/2406.06563v1","url_pdf":"https://arxiv.org/pdf/2406.06563v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"skywork-moe-a-deep-dive-into-training","repo_url":"https://github.com/sail-sg/adan","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.06563","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06563"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/adan","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f8c062b38dae0766","entry":"optimizer_kwargs","repo":"sail-sg/adan","repo_kind":"listed","path":"CV/timm/optim_factory.py","file_url":"https://github.com/sail-sg/adan/blob/HEAD/CV/timm/optim_factory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8c062b38dae0766"}},{"code_sha256_prefix":"1528f2fb366de84f","entry":"param_groups_layer_decay","repo":"sail-sg/adan","repo_kind":"listed","path":"CV/timm/optim_factory.py","file_url":"https://github.com/sail-sg/adan/blob/HEAD/CV/timm/optim_factory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1528f2fb366de84f"}},{"code_sha256_prefix":"88f4962784e09b88","entry":"param_groups_weight_decay","repo":"sail-sg/adan","repo_kind":"listed","path":"CV/timm/optim_factory.py","file_url":"https://github.com/sail-sg/adan/blob/HEAD/CV/timm/optim_factory.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88f4962784e09b88"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}