{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-centric-and-heterogeneity-adaptive","title":"FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism","arxiv_id":"2412.01523","date":"2024-12-02","proceeding":null,"authors":["Yujie Wang","Shiju Wang","Shenhan Zhu","Fangcheng Fu","Xinyi Liu","Xuefeng Xiao","Huixia Li","Jiashi Li","Faming Wu","Bin Cui"],"abstract":"Extending the context length (i.e., the maximum supported sequence length) of LLMs is of paramount significance. To facilitate long context training of LLMs, sequence parallelism has emerged as an essential technique, which scatters each input sequence across multiple devices and necessitates communication to process the sequence. In essence, existing sequence parallelism methods assume homogeneous sequence lengths (i.e., all input sequences are equal in length) and therefore leverages a single, static scattering strategy for all input sequences. However, in reality, the sequence lengths in LLM training corpora exhibit substantial variability, often following a long-tail distribution, which leads to workload heterogeneity. In this paper, we show that employing a single, static strategy results in inefficiency and resource under-utilization, highlighting the need for adaptive approaches to handle the heterogeneous workloads across sequences. To address this, we propose a heterogeneity-adaptive sequence parallelism method. For each training step, our approach captures the variability in sequence lengths and assigns the optimal combination of scattering strategies based on workload characteristics. We model this problem as a linear programming optimization and design an efficient and effective solver to find the optimal solution. Furthermore, we implement our method in a high-performance system that supports adaptive parallelization in distributed LLM training. Experimental results demonstrate that our system outperforms state-of-the-art training frameworks by up to 1.98x.","url_abs":"https://arxiv.org/abs/2412.01523v3","url_pdf":"https://arxiv.org/pdf/2412.01523v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-centric-and-heterogeneity-adaptive","repo_url":"https://github.com/pku-dair/hetu-galvatron","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.01523","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.01523"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pku-dair/hetu-galvatron","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c432b2807aeec9e0","entry":"galvatron_profile_args","repo":"pku-dair/hetu-galvatron","repo_kind":"listed","path":"galvatron/core/profiler/arguments.py","file_url":"https://github.com/pku-dair/hetu-galvatron/blob/HEAD/galvatron/core/profiler/arguments.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c432b2807aeec9e0"}},{"code_sha256_prefix":"01be37157b6581bc","entry":"galvatron_profile_hardware_args","repo":"pku-dair/hetu-galvatron","repo_kind":"listed","path":"galvatron/core/profiler/arguments.py","file_url":"https://github.com/pku-dair/hetu-galvatron/blob/HEAD/galvatron/core/profiler/arguments.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01be37157b6581bc"}},{"code_sha256_prefix":"e150d0d2237a0eea","entry":"get_time_cost_all_stages","repo":"pku-dair/hetu-galvatron","repo_kind":"listed","path":"galvatron/core/cost_model/cost_model_handler.py","file_url":"https://github.com/pku-dair/hetu-galvatron/blob/HEAD/galvatron/core/cost_model/cost_model_handler.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e150d0d2237a0eea"}},{"code_sha256_prefix":"5cefabd8436714e8","entry":"print_peak_memory","repo":"pku-dair/hetu-galvatron","repo_kind":"listed","path":"galvatron/core/profiler/utils.py","file_url":"https://github.com/pku-dair/hetu-galvatron/blob/HEAD/galvatron/core/profiler/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5cefabd8436714e8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}