{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dsp-dynamic-sequence-parallelism-for-multi","title":"DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers","arxiv_id":"2403.10266","date":"2024-03-15","proceeding":null,"authors":["Xuanlei Zhao","Shenggan Cheng","Chang Chen","Zangwei Zheng","Ziming Liu","Zheming Yang","Yang You"],"abstract":"Scaling multi-dimensional transformers to long sequences is indispensable across various domains. However, the challenges of large memory requirements and slow speeds of such sequences necessitate sequence parallelism. All existing approaches fall under the category of embedded sequence parallelism, which are limited to shard along a single sequence dimension, thereby introducing significant communication overhead. However, the nature of multi-dimensional transformers involves independent calculations across multiple sequence dimensions. To this end, we propose Dynamic Sequence Parallelism (DSP) as a novel abstraction of sequence parallelism. DSP dynamically switches the parallel dimension among all sequences according to the computation stage with efficient resharding strategy. DSP offers significant reductions in communication costs, adaptability across modules, and ease of implementation with minimal constraints. Experimental evaluations demonstrate DSP's superiority over state-of-the-art embedded sequence parallelism methods by remarkable throughput improvements ranging from 32.2% to 10x, with less than 25% communication volume.","url_abs":"https://arxiv.org/abs/2403.10266v5","url_pdf":"https://arxiv.org/pdf/2403.10266v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dsp-dynamic-sequence-parallelism-for-multi","repo_url":"https://github.com/nus-hpc-ai-lab/opendit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dsp-dynamic-sequence-parallelism-for-multi","repo_url":"https://github.com/NUS-HPC-AI-Lab/VideoSys","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.10266","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.10266"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nus-hpc-ai-lab/opendit","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NUS-HPC-AI-Lab/VideoSys","reach":null}],"summary":{"ran_draft_wrong":2,"ran":8,"ran_honours":1,"ran_violates":1,"unverified":7},"by_repo_kind":{"official":{"samples":18,"ran":12,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9fcdaa6e423e8aa7","entry":"Normalize","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/models/autoencoders/autoencoder_kl_open_sora_plan_v110.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/models/autoencoders/autoencoder_kl_open_sora_plan_v110.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9fcdaa6e423e8aa7"}},{"code_sha256_prefix":"32447bb9476ba9a4","entry":"all_to_all_comm","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/distributed/comm.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/distributed/comm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32447bb9476ba9a4"}},{"code_sha256_prefix":"c1b68cdd9e73316a","entry":"cast_tuple","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/models/autoencoders/autoencoder_kl_open_sora.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/models/autoencoders/autoencoder_kl_open_sora.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1b68cdd9e73316a"}},{"code_sha256_prefix":"f912c209e9c899ca","entry":"divisible_by","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/models/autoencoders/autoencoder_kl_open_sora.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/models/autoencoders/autoencoder_kl_open_sora.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f912c209e9c899ca"}},{"code_sha256_prefix":"6875a0fdda4fa123","entry":"get_distributed_init_method","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/engine/mp_utils.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/engine/mp_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6875a0fdda4fa123"}},{"code_sha256_prefix":"daeff11d0e2563e3","entry":"if_broadcast_cross","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/pab/pab_mgr.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/pab/pab_mgr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"daeff11d0e2563e3"}},{"code_sha256_prefix":"90d456a8c999fe52","entry":"if_broadcast_spatial","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/pab/pab_mgr.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/pab/pab_mgr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90d456a8c999fe52"}},{"code_sha256_prefix":"b1810f6a3b221de3","entry":"if_broadcast_temporal","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/pab/pab_mgr.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/pab/pab_mgr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1810f6a3b221de3"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/models/autoencoders/autoencoder_kl_open_sora_plan_v110.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/models/autoencoders/autoencoder_kl_open_sora_plan_v110.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"12c9db8fb79f543d","entry":"set_distributed_state","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/distributed/parallel_mgr.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/distributed/parallel_mgr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12c9db8fb79f543d"}},{"code_sha256_prefix":"4809422d25cbda07","entry":"split_sequence","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/distributed/comm.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/distributed/comm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4809422d25cbda07"}},{"code_sha256_prefix":"5d0218d6974725ed","entry":"tensor_to_video","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/models/autoencoders/autoencoder_kl_open_sora_plan_v110.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/models/autoencoders/autoencoder_kl_open_sora_plan_v110.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d0218d6974725ed"}},{"code_sha256_prefix":"24afe9113375d56e","entry":"DynamicParallelManager","repo":"NUS-HPC-AI-Lab/VideoSys","repo_kind":"listed","path":"videosys/core/distributed/parallel_mgr.py","file_url":"https://github.com/NUS-HPC-AI-Lab/VideoSys/blob/HEAD/videosys/core/distributed/parallel_mgr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"24afe9113375d56e"}},{"code_sha256_prefix":"686ce42a4e575400","entry":"add_timestamp","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/dcp/recompute.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/dcp/recompute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"686ce42a4e575400"}},{"code_sha256_prefix":"17f768f6ac1bbb38","entry":"auto_recompute","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/dcp/recompute.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/dcp/recompute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17f768f6ac1bbb38"}},{"code_sha256_prefix":"40ebfee15a7cbaa6","entry":"gather_sequence","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/distributed/comm.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/distributed/comm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40ebfee15a7cbaa6"}},{"code_sha256_prefix":"be3706734550c9d9","entry":"generate","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"gradio/cogvideox.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/gradio/cogvideox.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be3706734550c9d9"}},{"code_sha256_prefix":"945442cc2f6a19c9","entry":"is_odd","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/models/autoencoders/autoencoder_kl_open_sora.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/models/autoencoders/autoencoder_kl_open_sora.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"945442cc2f6a19c9"}},{"code_sha256_prefix":"73301f9777f1a2c0","entry":"recompute_func","repo":"nus-hpc-ai-lab/opendit","repo_kind":"official","path":"videosys/core/dcp/recompute.py","file_url":"https://github.com/nus-hpc-ai-lab/opendit/blob/HEAD/videosys/core/dcp/recompute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73301f9777f1a2c0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}