{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-parallelization-layouts-for-large","title":"Efficient Parallelization Layouts for Large-Scale Distributed Model Training","arxiv_id":"2311.05610","date":"2023-11-09","proceeding":null,"authors":["Johannes Hagemann","Samuel Weinbach","Konstantin Dobler","Maximilian Schall","Gerard de Melo"],"abstract":"Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, many of these strategies have complex interactions regarding the final training efficiency. Prior work tackling this problem did not have access to the latest set of optimizations, such as FlashAttention or sequence parallelism. In this work, we conduct a comprehensive ablation study of possible training configurations for large language models. We distill this large study into several key recommendations for the most efficient training. For instance, we find that using a micro-batch size of 1 usually enables the most efficient training layouts. Larger micro-batch sizes necessitate activation checkpointing or higher degrees of model parallelism and also lead to larger pipeline bubbles. Our most efficient configurations enable us to achieve state-of-the-art training efficiency results over a range of model sizes, most notably a Model FLOPs utilization of 70.5% when training a Llama 13B model.","url_abs":"https://arxiv.org/abs/2311.05610v3","url_pdf":"https://arxiv.org/pdf/2311.05610v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-parallelization-layouts-for-large","repo_url":"https://github.com/aleph-alpha/neurips-want-submission-efficient-parallelization-layouts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.05610","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.05610"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aleph-alpha/neurips-want-submission-efficient-parallelization-layouts","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"e04f771d7dd70023","entry":"preprocess","repo":"aleph-alpha/neurips-want-submission-efficient-parallelization-layouts","repo_kind":"official","path":"reproduce_figure_model_vs_pipe_parallel.py","file_url":"https://github.com/aleph-alpha/neurips-want-submission-efficient-parallelization-layouts/blob/HEAD/reproduce_figure_model_vs_pipe_parallel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e04f771d7dd70023"}},{"code_sha256_prefix":"404867587bfa9018","entry":"scale_y_axis_labels","repo":"aleph-alpha/neurips-want-submission-efficient-parallelization-layouts","repo_kind":"official","path":"reproduce_figure_checkpointing.py","file_url":"https://github.com/aleph-alpha/neurips-want-submission-efficient-parallelization-layouts/blob/HEAD/reproduce_figure_checkpointing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"404867587bfa9018"}},{"code_sha256_prefix":"7f664dc394bb3883","entry":"format_axes","repo":"aleph-alpha/neurips-want-submission-efficient-parallelization-layouts","repo_kind":"official","path":"latexify.py","file_url":"https://github.com/aleph-alpha/neurips-want-submission-efficient-parallelization-layouts/blob/HEAD/latexify.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7f664dc394bb3883"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}