</> Code with Papers by Syntology

Methods › General › Stochastic Optimization › RAdam

RAdam

65 papers tagged archive 2025-07-28

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

Rectified Adam, or RAdam, is a variant of the Adam stochastic optimizer that introduces a term to rectify the variance of the adaptive learning rate. It seeks to tackle the bad convergence problem suffered by Adam. The authors argue that the root cause of this behaviour is that the adaptive learning rate has undesirably large variance in the early stage of model training, due to the limited amount of training samples being used. Thus, to reduce such variance, it is better to use smaller learning rates in the first few epochs of training - which justifies the warmup heuristic. This heuristic motivates RAdam which rectifies the variance problem:

gₜ = ∇_θfₜ(θₜ₋₁)

vₜ = 1/β₂vₜ₋₁ + (1-β₂)g²ₜ

mₜ = β₁mₜ₋₁ + (1-β₁)gₜ

m̂ₜ̂ = mₜ / (1-βᵗ₁)

ρₜ = ρ_∞ - 2tβᵗ₂/(1-βᵗ₂)

ρ_∞ = 2/(1-β₂) - 1

If the variance is tractable - ρₜ > 4 then:

...the adaptive learning rate is computed as:

lₜ = √((1-βᵗ₂)/vₜ)

...the variance rectification term is calculated as:

rₜ = √(((ρₜ-4)(ρₜ-2)ρ_∞)/((ρ_∞-4)(ρ_∞-2)ρₜ))

...and we update parameters with adaptive momentum:

θₜ = θₜ₋₁ - αₜrₜm̂ₜlₜ

If the variance isn't tractable we update instead with:

θₜ = θₜ₋₁ - αₜm̂ₜ

Source: On the Variance of the Adaptive Learning Rate and BeyondSee Code · pytorch/pytorch

Papers archive 2025-07-28

30 shown of 65, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.

Tasks archive 2025-07-28

20 shown of 73 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

TaskPapers
Language Modelling4
Deep Learning3
Image Classification3
Language Modeling3
Articles2
Autonomous Driving2
BIG-bench Machine Learning2
Classification2
General Classification2
Machine Translation2
Translation2
image-classification2
Anomaly Detection1
Attribute1
Autonomous Vehicles1
CPU1
Claim Verification1
Continual Learning1
Decoder1
DeepFake Detection1

Usage over time archive 2025-07-28

Papers per year tagged with RAdam: 2019 to 2025, peak 19 19 0 2019: 2 papers 2019 2020: 6 papers 2020 2021: 7 papers 2021 2022: 11 papers 2022 2023: 19 papers 2023 2024: 10 papers 2024 2025: 10 papers 2025
Papers per year the archive tags with this method, by the paper's archive date (65 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Stochastic Optimization

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections