Methods › General › Data Parallel Methods
Data Parallel Methods
The archive attaches this collection's text per method and the copies differ: 3 distinct texts across 14 of the 15 methods here. All are shown, most-carried first (a tie goes to the text carrying Papers with Code's collection boilerplate, then to the longer text); no vote is taken between them.
Text 1, carried by 10 of 15 methods:
This section contains a compilation of distributed methods for scaling deep learning to very large models. There are many different strategies for scaling training across multiple devices, including:
-
Data Parallel : for each node we use the same model parameters to do forward propagation, but we send a small batch of different data to each node, compute the gradient normally, and send it back to the main node. Once we have all the gradients, we calculate the weighted average and use this to update the model parameters.
-
Model Parallel : for each node we assign different layers to it. During forward propagation, we start in the node with the first layers, then move onto the next, and so on. Once forward propagation is done we calculate gradients for the last node, and update model parameters for that node. Then we backpropagate onto the penultimate node, update the parameters, and so on.
-
Additional methods including Hybrid Parallel, Auto Parallel, and Distributed Communication.
Image credit: Jordi Torres.
Text 2, carried by 2 of 15 methods:
Stochastic Optimization methods are used to optimize neural networks. We typically take a mini-batch of data, hence 'stochastic', and perform a type of gradient descent with this minibatch. Below you can find a continuously updating list of stochastic optimization algorithms.
Text 3, carried by 2 of 15 methods:
This section contains a compilation of distributed data parallel methods for deep learning. For each node we use the same model parameters to do forward propagation, but we send a small batch of different data to each node, compute the gradient normally, and send it back to the main node. Once we have all the gradients, we calculate the weighted average and use this to update the model parameters.
Image credit: Jordi Torres.
Methods
All 15 methods in this collection, most-tagged first. Year is the archive's introduced_year; the archive stores 2000 when it has none, shown here as “–”. Papers counts distinct papers the archive tags with the method. Click a heading to sort.
| Local SGD | – | 69 |
| Gradient Sparsification | – | 38 |
| ZeRO | – | 9 |
| Accordion | – | 4 |
| ZeRO-Offload | – | 4 |
| PowerSGD | – | 3 |
| PyTorch DDP | – | 3 |
| Crossbow | – | 2 |
| ZeRO-Infinity | – | 2 |
| ALQ and AMQ Gradient Quantization with Adaptive Levels/Multiplier | – | 1 |
| BAGUA | – | 1 |
| ByteScheduler | 2019 | 1 |
| DABMD Distributed Any-Batch Mirror Descent | 2020 | 1 |
| NUQSGD Nonuniform Quantization for Stochastic Gradient Descent | – | 1 |
| Wavelet Distributed Training | 2021 | 0 |