{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-11686","title":"LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training","arxiv_id":"2602.11686","date":"2026-02-12","proceeding":null,"authors":["Xinyi Liu","Yujie Wang","Fangcheng Fu","Xuefeng Xiao","Huixia Li","Jiashi Li","Bin Cui"],"abstract":"Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different input data. However, during expert parallel training, dynamic routing results in significant load imbalance among experts: a handful of overloaded experts hinder overall iteration, emerging as a training bottleneck. In this paper, we introduce LAER-MoE, an efficient MoE training framework. The core of LAER-MoE is a novel parallel paradigm, Fully Sharded Expert Parallel (FSEP), which fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication during training. This allows for flexible re-layout of expert parameters during training to enhance load balancing. In particular, we perform fine-grained scheduling of communication operations to minimize communication overhead. Additionally, we develop a load balancing planner to formulate re-layout strategies of experts and routing schemes for tokens during training. We perform experiments on an A100 cluster, and the results indicate that our system achieves up to 1.69x acceleration compared to the current state-of-the-art training systems. Source code available at https://github.com/PKU-DAIR/Hetu-Galvatron/tree/laer-moe.","url_abs":"https://arxiv.org/abs/2602.11686","url_pdf":"https://arxiv.org/pdf/2602.11686","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.11686","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.11686"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/PKU-DAIR/Hetu-Galvatron","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c432b2807aeec9e0","entry":"galvatron_profile_args","repo":"PKU-DAIR/Hetu-Galvatron","repo_kind":"found_in_text","path":"galvatron/core/profiler/arguments.py","file_url":"https://github.com/PKU-DAIR/Hetu-Galvatron/blob/HEAD/galvatron/core/profiler/arguments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c432b2807aeec9e0"}},{"code_sha256_prefix":"01be37157b6581bc","entry":"galvatron_profile_hardware_args","repo":"PKU-DAIR/Hetu-Galvatron","repo_kind":"found_in_text","path":"galvatron/core/profiler/arguments.py","file_url":"https://github.com/PKU-DAIR/Hetu-Galvatron/blob/HEAD/galvatron/core/profiler/arguments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01be37157b6581bc"}},{"code_sha256_prefix":"e150d0d2237a0eea","entry":"get_time_cost_all_stages","repo":"PKU-DAIR/Hetu-Galvatron","repo_kind":"found_in_text","path":"galvatron/core/cost_model/cost_model_handler.py","file_url":"https://github.com/PKU-DAIR/Hetu-Galvatron/blob/HEAD/galvatron/core/cost_model/cost_model_handler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e150d0d2237a0eea"}},{"code_sha256_prefix":"5cefabd8436714e8","entry":"print_peak_memory","repo":"PKU-DAIR/Hetu-Galvatron","repo_kind":"found_in_text","path":"galvatron/core/profiler/utils.py","file_url":"https://github.com/PKU-DAIR/Hetu-Galvatron/blob/HEAD/galvatron/core/profiler/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5cefabd8436714e8"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}