Methods › General › Stochastic Optimization › AMSBound
AMSBound
Introduced by Liangchen Luo et al. in Adaptive Gradient Methods with Dynamic Bound of Learning Rate
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AMSBound is a variant of the AMSGrad stochastic optimizer which is designed to be more robust to extreme learning rates. Dynamic bounds are employed on learning rates, where the lower and upper bound are initialized as zero and infinity respectively, and they both smoothly converge to a constant final step size. AMSBound can be regarded as an adaptive method at the beginning of training, and it gradually and smoothly transforms to SGD (or with momentum) as time step increases.
gₜ = ∇fₜ(xₜ)
mₜ = β₁ₜmₜ₋₁ + (1-β₁ₜ)gₜ
vₜ = β₂vₜ₋₁ + (1-β₂)gₜ²
v̂ₜ = max(v̂ₜ₋₁, vₜ) and Vₜ = diag(v̂ₜ)
η= Clip(α/√(Vₜ), ηₗ(t), ηᵤ(t)) and ηₜ = η/√(t)
xₜ₊₁ = Π_(ℱ, diag(ηₜ⁻¹))(xₜ - ηₜ ⊙mₜ )
Where α is the initial step size, and ηₗ and ηᵤ are the lower and upper bound functions respectively.
Papers archive 2025-07-28
1 shown of 1, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Adaptive Gradient Methods with Dynamic Bound of Learning Rate 26 Feb 2019 · 5 repositories · arXiv:1902.09843
Tasks archive 2025-07-28
The archive attaches no task to a paper tagged with this method.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections