Methods › General › Stochastic Optimization › AMSGrad
AMSGrad
Introduced by Sashank J. Reddi et al. in On the Convergence of Adam and Beyond
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AMSGrad is a stochastic optimization method that seeks to fix a convergence issue with Adam based optimizers. AMSGrad uses the maximum of past squared gradients vₜ rather than the exponential average to update the parameters:
mₜ = β₁mₜ₋₁ + (1-β₁)gₜ
vₜ = β₂vₜ₋₁ + (1-β₂)gₜ²
v̂ₜ = max(v̂ₜ₋₁, vₜ)
θₜ₊₁ = θₜ - η/(√(v̂ₜ) + ϵ)mₜ
Papers archive 2025-07-28
30 shown of 49, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation 14 Oct 2024 · 0 repositories · arXiv:2410.10533
-
MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence 24 May 2024 · 1 repository · arXiv:2405.15593Syntology ran 7 of 10 samples · 3 unverified
-
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent 17 Jan 2024 · 0 repositories · arXiv:2401.08893
-
FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data 18 Sep 2023 · 0 repositories · arXiv:2309.09719
-
Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods 21 May 2023 · 0 repositories · arXiv:2305.12475
-
UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization 9 May 2023 · 0 repositories · arXiv:2305.05675
-
𝒞ᵏ-continuous Spline Approximation with TensorFlow Gradient Descent Optimizers 22 Mar 2023 · 0 repositories · arXiv:2303.12454
-
AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks 1 Mar 2023 · 0 repositories · arXiv:2303.00565
-
Optimization Methods in Deep Learning: A Comprehensive Overview 19 Feb 2023 · 0 repositories · arXiv:2302.09566
-
Langevin dynamics based algorithm e-THεO POULA for stochastic optimization problems with discontinuous stochastic gradient 24 Oct 2022 · 1 repository · arXiv:2210.13193
-
Communication-Efficient Adam-Type Algorithms for Distributed Data Mining 14 Oct 2022 · 0 repositories · arXiv:2210.07454
-
Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization 1 Jun 2022 · 0 repositories · arXiv:2206.00743
-
On Distributed Adaptive Optimization with Gradient Compression 11 May 2022 · 0 repositories · arXiv:2205.05632
-
AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization 18 Jan 2022 · 1 repository · arXiv:2201.06714
-
Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates 5 Jan 2022 · 1 repository · arXiv:2201.01652
-
A Novel Convergence Analysis for Algorithms of the Adam Family 7 Dec 2021 · 0 repositories · arXiv:2112.03459
-
Convergence of adaptive algorithms for constrained weakly convex optimization 1 Dec 2021 · 0 repositories
-
Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization 1 Nov 2021 · 0 repositories · arXiv:2111.00705
-
SGD Can Converge to Local Maxima 29 Sep 2021 · 0 repositories
-
On the Convergence of Decentralized Adaptive Gradient Methods 7 Sep 2021 · 0 repositories · arXiv:2109.03194
-
Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR Prediction 30 Jul 2021 · 2 repositories · arXiv:2107.14432
-
SGD with a Constant Large Learning Rate Can Converge to Local Maxima 25 Jul 2021 · 0 repositories · arXiv:2107.11774
-
Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function 19 Jul 2021 · 1 repository · arXiv:2107.08649
-
A New Adaptive Gradient Method with Gradient Decomposition 18 Jul 2021 · 0 repositories · arXiv:2107.08377
-
A decreasing scaling transition scheme from Adam to SGD 12 Jun 2021 · 2 repositories · arXiv:2106.06749
-
Escaping Saddle Points Faster with Stochastic Momentum 5 Jun 2021 · 0 repositories · arXiv:2106.02985
-
Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator 30 Apr 2021 · 0 repositories · arXiv:2104.14840
-
Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series 25 Jan 2021 · 0 repositories · arXiv:2101.10037
-
Adam revisited: a weighted past gradients perspective 1 Jan 2021 · 0 repositories · arXiv:2101.00238
-
Adaptive Optimizers with Sparse Group Lasso 1 Jan 2021 · 0 repositories
Tasks archive 2025-07-28
20 shown of 39 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections