Papers › Group-based Interleaved Pipeline Parallelism for Large-scale DNN Training
Group-based Interleaved Pipeline Parallelism for Large-scale DNN Training
Pengcheng Yang, XiaoMing Zhang, Wenpeng Zhang, Ming Yang, Hong Wei
The recent trend of using large-scale deep neural networks (DNN) to boost performance has propelled the development of the parallel pipelining technique for efficient DNN training, which has resulted in the development of several prominent pipelines such as GPipe, PipeDream, and PipeDream-2BW. However, the current leading pipeline, PipeDream-2BW, still suffers from two major drawbacks, namely the excessive memory redundancy and the delayed weight updates across all stages. In this work, we propose a novel pipeline named WPipe, which achieves better memory efficiency and fresher weight updates. WPipe uses a novel pipelining scheme, which divides model partitions into two groups. It moves the forward pass of the next period to the front of the backward pass of the current period in the first group, retains the order in the second group, and updates each group alternatively. This will eliminate half of the delayed gradients and memory redundancy compared to PipeDream-2BW. The results of our experiments, in which large BERT language models were trained, show that compared to PipeDream-2BW, WPipe achieves 1.4× acceleration, reduces the memory footprint by 36% with similar final model accuracy.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections