Methods › General › Stochastic Optimization › AMSGrad
AMSGrad
Introduced by Sashank J. Reddi et al. in On the Convergence of Adam and Beyond
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AMSGrad is a stochastic optimization method that seeks to fix a convergence issue with Adam based optimizers. AMSGrad uses the maximum of past squared gradients vₜ rather than the exponential average to update the parameters:
mₜ = β₁mₜ₋₁ + (1-β₁)gₜ
vₜ = β₂vₜ₋₁ + (1-β₂)gₜ²
v̂ₜ = max(v̂ₜ₋₁, vₜ)
θₜ₊₁ = θₜ - η/(√(v̂ₜ) + ϵ)mₜ
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: Q-AMSGrad.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 49 · with a code link: 14 · where Syntology ran a sample: 4 (4 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (4 of 49 tagged: 4 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 49 of 49, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation 14 Oct 2024 · 0 repositories · arXiv:2410.10533
-
MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence 24 May 2024 · 1 repository · arXiv:2405.15593Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 10 harvested samples)
-
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent 17 Jan 2024 · 0 repositories · arXiv:2401.08893
-
FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data 18 Sep 2023 · 0 repositories · arXiv:2309.09719
-
Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods 21 May 2023 · 0 repositories · arXiv:2305.12475
-
UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization 9 May 2023 · 0 repositories · arXiv:2305.05675
-
𝒞ᵏ-continuous Spline Approximation with TensorFlow Gradient Descent Optimizers 22 Mar 2023 · 0 repositories · arXiv:2303.12454
-
AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks 1 Mar 2023 · 0 repositories · arXiv:2303.00565
-
Optimization Methods in Deep Learning: A Comprehensive Overview 19 Feb 2023 · 0 repositories · arXiv:2302.09566
-
Langevin dynamics based algorithm e-THεO POULA for stochastic optimization problems with discontinuous stochastic gradient 24 Oct 2022 · 1 repository · arXiv:2210.13193
-
Communication-Efficient Adam-Type Algorithms for Distributed Data Mining 14 Oct 2022 · 0 repositories · arXiv:2210.07454
-
Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization 1 Jun 2022 · 0 repositories · arXiv:2206.00743
-
On Distributed Adaptive Optimization with Gradient Compression 11 May 2022 · 0 repositories · arXiv:2205.05632
-
AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization 18 Jan 2022 · 1 repository · arXiv:2201.06714
-
Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates 5 Jan 2022 · 1 repository · arXiv:2201.01652
-
A Novel Convergence Analysis for Algorithms of the Adam Family 7 Dec 2021 · 0 repositories · arXiv:2112.03459
-
Convergence of adaptive algorithms for constrained weakly convex optimization 1 Dec 2021 · 0 repositories
-
Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization 1 Nov 2021 · 0 repositories · arXiv:2111.00705
-
SGD Can Converge to Local Maxima 29 Sep 2021 · 0 repositories
-
On the Convergence of Decentralized Adaptive Gradient Methods 7 Sep 2021 · 0 repositories · arXiv:2109.03194
-
Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR Prediction 30 Jul 2021 · 2 repositories · arXiv:2107.14432
-
SGD with a Constant Large Learning Rate Can Converge to Local Maxima 25 Jul 2021 · 0 repositories · arXiv:2107.11774
-
Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function 19 Jul 2021 · 1 repository · arXiv:2107.08649
-
A New Adaptive Gradient Method with Gradient Decomposition 18 Jul 2021 · 0 repositories · arXiv:2107.08377
-
A decreasing scaling transition scheme from Adam to SGD 12 Jun 2021 · 2 repositories · arXiv:2106.06749
-
Escaping Saddle Points Faster with Stochastic Momentum 5 Jun 2021 · 0 repositories · arXiv:2106.02985
-
Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator 30 Apr 2021 · 0 repositories · arXiv:2104.14840
-
Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series 25 Jan 2021 · 0 repositories · arXiv:2101.10037
-
Adam revisited: a weighted past gradients perspective 1 Jan 2021 · 0 repositories · arXiv:2101.00238
-
Adaptive Optimizers with Sparse Group Lasso 1 Jan 2021 · 0 repositories
-
On the Marginal Regret Bound Minimization of Adaptive Methods 1 Jan 2021 · 0 repositories
-
CADA: Communication-Adaptive Distributed Adam 31 Dec 2020 · 1 repository · arXiv:2012.15469Syntology 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model 29 Sep 2020 · 0 repositories · arXiv:2009.13799
-
Towards Deep Robot Learning with Optimizer applicable to Non-stationary Problems 31 Jul 2020 · 0 repositories · arXiv:2007.15890
-
Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent 15 Jul 2020 · 0 repositories · arXiv:2007.07422
-
MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients 21 Jun 2020 · 1 repository · arXiv:2006.11918
-
Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search) 11 Jun 2020 · 1 repository · arXiv:2006.06835Syntology 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Convergence of adaptive algorithms for weakly convex constrained optimization 11 Jun 2020 · 0 repositories · arXiv:2006.06650
-
MixML: A Unified Analysis of Weakly Consistent Parallel Learning 14 May 2020 · 0 repositories · arXiv:2005.06706
-
Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling 15 Feb 2020 · 0 repositories · arXiv:2002.06286
-
On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods 17 Jan 2020 · 0 repositories · arXiv:2001.06130
-
An Adaptive and Momental Bound Method for Stochastic Learning 27 Oct 2019 · 2 repositories · arXiv:1910.12249Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
diffGrad: An Optimization Method for Convolutional Neural Networks 12 Sep 2019 · 1 repository · arXiv:1909.11015
-
Meta-descent for Online, Continual Prediction 17 Jul 2019 · 0 repositories · arXiv:1907.07751
-
Power Gradient Descent 11 Jun 2019 · 0 repositories · arXiv:1906.04787
-
Multivariate, Multistep Forecasting, Reconstruction and Feature Selection of Ocean Waves via Recurrent and Sequence-to-Sequence Networks 1 Jun 2019 · 1 repository · arXiv:1906.00195
-
Accelerating first order optimization algorithms 1 May 2019 · 0 repositories
-
Optimistic Acceleration for Optimization 1 May 2019 · 0 repositories
-
On the Convergence of Adam and Beyond 19 Apr 2019 · 3 repositories · arXiv:1904.09237Syntology 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
Tasks archive 2025-07-28
20 shown of 39 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections