Methods › General › Stochastic Optimization › AdaMod
AdaMod
Introduced by Jianbang Ding et al. in An Adaptive and Momental Bound Method for Stochastic Learning
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AdaMod is a stochastic optimizer that restricts adaptive learning rates with adaptive and momental upper bounds. The dynamic learning rate bounds are based on the exponential moving averages of the adaptive learning rates themselves, which smooth out unexpected large learning rates and stabilize the training of deep neural networks.
The weight updates are performed as:
gₜ = ∇fₜ(θₜ₋₁)
mₜ = β₁mₜ₋₁ + (1-β₁)gₜ
vₜ = β₂vₜ₋₁ + (1-β₂)gₜ²
m̂ₜ = mₜ / (1 - βᵗ₁)
v̂ₜ = vₜ / (1 - βᵗ₂)
ηₜ = αₜ / (√(v̂ₜ) + ϵ)
sₜ = β₃sₜ₋₁ + (1-β₃)ηₜ
η̂ₜ = min(ηₜ, sₜ)
θₜ = θₜ₋₁ - η̂ₜm̂ₜ
Papers archive 2025-07-28
1 shown of 1, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
An Adaptive and Momental Bound Method for Stochastic Learning 27 Oct 2019 · 2 repositories · arXiv:1910.12249Syntology ran 0 of 2 samples · 2 unverified
Tasks archive 2025-07-28
1 task the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Stochastic Optimization | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections