{"url":"/method/amsgrad","slug":"amsgrad","name":"AMSGrad","full_name":"AMSGrad","full_name_withheld":false,"description_markdown":"**AMSGrad** is a stochastic optimization method that seeks to fix a convergence issue with [Adam](https://paperswithcode.com/method/adam) based optimizers. AMSGrad uses the maximum of past squared gradients \r\n$v\\_{t}$ rather than the exponential average to update the parameters:\r\n\r\n$$m\\_{t} = \\beta\\_{1}m\\_{t-1} + \\left(1-\\beta\\_{1}\\right)g\\_{t} $$\r\n\r\n$$v\\_{t} = \\beta\\_{2}v\\_{t-1} + \\left(1-\\beta\\_{2}\\right)g\\_{t}^{2}$$\r\n\r\n$$ \\hat{v}\\_{t} = \\max\\left(\\hat{v}\\_{t-1}, v\\_{t}\\right) $$\r\n\r\n$$\\theta\\_{t+1} = \\theta\\_{t} - \\frac{\\eta}{\\sqrt{\\hat{v}_{t}} + \\epsilon}m\\_{t}$$","description_state":"present","introduced_year":null,"introduced_by":{"title":"On the Convergence of Adam and Beyond","paper":"/paper/on-the-convergence-of-adam-and-beyond-1","first_author":"Sashank J. Reddi","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/on-the-convergence-of-adam-and-beyond-1"},"source":{"url":"http://arxiv.org/abs/1904.09237v1","title":"On the Convergence of Adam and Beyond","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/b7bda236d18815052378c88081f64935427d7716/torch/optim/adam.py#L6","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":49,"archive_num_papers":49,"papers_newest_first":[{"paper":null,"title":"Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation","date":"2024-10-14","arxiv_id":"2410.10533","n_code_links":0,"syntology":null},{"paper":"/paper/microadam-accurate-adaptive-optimization-with","title":"MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence","date":"2024-05-24","arxiv_id":"2405.15593","n_code_links":1,"syntology":{"ran":7,"of":10,"unverified":3,"pointer_only":0}},{"paper":null,"title":"MADA: Meta-Adaptive Optimizers through hyper-gradient Descent","date":"2024-01-17","arxiv_id":"2401.08893","n_code_links":0,"syntology":null},{"paper":null,"title":"FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data","date":"2023-09-18","arxiv_id":"2309.09719","n_code_links":0,"syntology":null},{"paper":null,"title":"Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods","date":"2023-05-21","arxiv_id":"2305.12475","n_code_links":0,"syntology":null},{"paper":null,"title":"UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization","date":"2023-05-09","arxiv_id":"2305.05675","n_code_links":0,"syntology":null},{"paper":null,"title":"$\\mathcal{C}^k$-continuous Spline Approximation with TensorFlow Gradient Descent Optimizers","date":"2023-03-22","arxiv_id":"2303.12454","n_code_links":0,"syntology":null},{"paper":null,"title":"AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks","date":"2023-03-01","arxiv_id":"2303.00565","n_code_links":0,"syntology":null},{"paper":null,"title":"Optimization Methods in Deep Learning: A Comprehensive Overview","date":"2023-02-19","arxiv_id":"2302.09566","n_code_links":0,"syntology":null},{"paper":"/paper/langevin-dynamics-based-algorithm-e-th","title":"Langevin dynamics based algorithm e-TH$\\varepsilon$O POULA for stochastic optimization problems with discontinuous stochastic gradient","date":"2022-10-24","arxiv_id":"2210.13193","n_code_links":1,"syntology":null},{"paper":null,"title":"Communication-Efficient Adam-Type Algorithms for Distributed Data Mining","date":"2022-10-14","arxiv_id":"2210.07454","n_code_links":0,"syntology":null},{"paper":null,"title":"Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization","date":"2022-06-01","arxiv_id":"2206.00743","n_code_links":0,"syntology":null},{"paper":null,"title":"On Distributed Adaptive Optimization with Gradient Compression","date":"2022-05-11","arxiv_id":"2205.05632","n_code_links":0,"syntology":null},{"paper":"/paper/adaterm-adaptive-t-distribution-estimated","title":"AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization","date":"2022-01-18","arxiv_id":"2201.06714","n_code_links":1,"syntology":null},{"paper":"/paper/convergence-and-complexity-of-stochastic","title":"Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates","date":"2022-01-05","arxiv_id":"2201.01652","n_code_links":1,"syntology":null},{"paper":null,"title":"A Novel Convergence Analysis for Algorithms of the Adam Family","date":"2021-12-07","arxiv_id":"2112.03459","n_code_links":0,"syntology":null},{"paper":null,"title":"Convergence of adaptive algorithms for constrained weakly convex optimization","date":"2021-12-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization","date":"2021-11-01","arxiv_id":"2111.00705","n_code_links":0,"syntology":null},{"paper":null,"title":"SGD Can Converge to Local Maxima","date":"2021-09-29","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"On the Convergence of Decentralized Adaptive Gradient Methods","date":"2021-09-07","arxiv_id":"2109.03194","n_code_links":0,"syntology":null},{"paper":"/paper/adaptive-optimizers-with-sparse-group-lasso-1","title":"Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR Prediction","date":"2021-07-30","arxiv_id":"2107.14432","n_code_links":2,"syntology":null},{"paper":null,"title":"SGD with a Constant Large Learning Rate Can Converge to Local Maxima","date":"2021-07-25","arxiv_id":"2107.11774","n_code_links":0,"syntology":null},{"paper":"/paper/non-asymptotic-estimates-for-tusla-algorithm","title":"Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function","date":"2021-07-19","arxiv_id":"2107.08649","n_code_links":1,"syntology":null},{"paper":null,"title":"A New Adaptive Gradient Method with Gradient Decomposition","date":"2021-07-18","arxiv_id":"2107.08377","n_code_links":0,"syntology":null},{"paper":"/paper/decreasing-scaling-transition-from-adaptive","title":"A decreasing scaling transition scheme from Adam to SGD","date":"2021-06-12","arxiv_id":"2106.06749","n_code_links":2,"syntology":null},{"paper":null,"title":"Escaping Saddle Points Faster with Stochastic Momentum","date":"2021-06-05","arxiv_id":"2106.02985","n_code_links":0,"syntology":null},{"paper":null,"title":"Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator","date":"2021-04-30","arxiv_id":"2104.14840","n_code_links":0,"syntology":null},{"paper":null,"title":"Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series","date":"2021-01-25","arxiv_id":"2101.10037","n_code_links":0,"syntology":null},{"paper":null,"title":"Adam revisited: a weighted past gradients perspective","date":"2021-01-01","arxiv_id":"2101.00238","n_code_links":0,"syntology":null},{"paper":null,"title":"Adaptive Optimizers with Sparse Group Lasso","date":"2021-01-01","arxiv_id":null,"n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":12},{"task":"/task/deep-learning","name":"Deep Learning","papers":3},{"task":"/task/distributed-optimization","name":"Distributed Optimization","papers":3},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":2},{"task":"/task/bilevel-optimization","name":"Bilevel Optimization","papers":2},{"task":"/task/open-question","name":"Open-Ended Question Answering","papers":2},{"task":"/task/quantization","name":"Quantization","papers":2},{"task":"/task/scheduling","name":"Scheduling","papers":2},{"task":"/task/time-series-1","name":"Time Series","papers":2},{"task":"/task/time-series","name":"Time Series Analysis","papers":2},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":2},{"task":"/task/type","name":"Vocal Bursts Type Prediction","papers":2},{"task":"/task/atari-games","name":"Atari Games","papers":1},{"task":"/task/binary-classification","name":"Binary Classification","papers":1},{"task":"/task/click-through-rate-prediction","name":"Click-Through Rate Prediction","papers":1},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":1},{"task":"/task/dictionary-learning","name":"Dictionary Learning","papers":1},{"task":"/task/distributed-computing","name":"Distributed Computing","papers":1},{"task":"/task/federated-learning","name":"Federated Learning","papers":1},{"task":"/task/hyperparameter-optimization","name":"Hyperparameter Optimization","papers":1}],"tasks_shown":20,"n_tasks":39,"usage_by_year":[{"year":"2019","papers":8},{"year":"2020","papers":10},{"year":"2021","papers":16},{"year":"2022","papers":6},{"year":"2023","papers":6},{"year":"2024","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/amsgrad"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}