{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/megascale-scaling-large-language-model","title":"MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs","arxiv_id":"2402.15627","date":"2024-02-23","proceeding":null,"authors":["Ziheng Jiang","Haibin Lin","Yinmin Zhong","Qi Huang","Yangrui Chen","Zhi Zhang","Yanghua Peng","Xiang Li","Cong Xie","Shibiao Nong","Yulu Jia","Sun He","Hongmin Chen","Zhihao Bai","Qi Hou","Shipeng Yan","Ding Zhou","Yiyao Sheng","Zhuo Jiang","Haohan Xu","Haoran Wei","Zhang Zhang","Pengfei Nie","Leqi Zou","Sida Zhao","Liang Xiang","Zherui Liu","Zhe Li","Xiaoying Jia","Jianxi Ye","Xin Jin","Xin Liu"],"abstract":"We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research.","url_abs":"https://arxiv.org/abs/2402.15627v1","url_pdf":"https://arxiv.org/pdf/2402.15627v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"megascale-scaling-large-language-model","repo_url":"https://github.com/volcengine/vescale","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.15627","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.15627"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/volcengine/vescale","reach":null}],"summary":{"ran_honours":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e4c082ad08de64d9","entry":"estimate_llama2","repo":"volcengine/vescale","repo_kind":"official","path":"legacy/examples/llama2_4D_finetune/llama_train.py","file_url":"https://github.com/volcengine/vescale/blob/HEAD/legacy/examples/llama2_4D_finetune/llama_train.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4c082ad08de64d9"}},{"code_sha256_prefix":"e3beedb28aed150b","entry":"estimate_mixtral","repo":"volcengine/vescale","repo_kind":"official","path":"legacy/examples/mixtral_4D_benchmark/mixtral_train.py","file_url":"https://github.com/volcengine/vescale/blob/HEAD/legacy/examples/mixtral_4D_benchmark/mixtral_train.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3beedb28aed150b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}