Methods › General › Learning Rate Schedules
Learning Rate Schedules
Learning Rate Schedules refer to schedules for the learning rate during the training of neural networks. Below you can find a continuously updating list of learning rate schedules.
Methods
All 13 methods in this collection, most-tagged first. Year is the archive's introduced_year; the archive stores 2000 when it has none, shown here as “–”. Papers counts distinct papers the archive tags with the method. Click a heading to sort.
| Linear Warmup With Linear Decay | – | 7,076 |
| Cosine Annealing | – | 3,965 |
| Linear Warmup With Cosine Annealing | – | 3,797 |
| Inverse Square Root Schedule | – | 702 |
| Exponential Decay | – | 110 |
| Step Decay | – | 73 |
| Slanted Triangular Learning Rates | – | 40 |
| Linear Warmup | – | 38 |
| 1cycle 1cycle learning rate scheduling policy | – | 13 |
| Polynomial Rate Decay | – | 10 |
| Cosine Power Annealing | – | 6 |
| Cyclical Learning Rate Policy | – | 5 |
| Log Decay Cyclical Log Annealing as a Learning Rate Scheduler | – | 0 |