{"url":"/method/nesterov-accelerated-gradient","slug":"nesterov-accelerated-gradient","name":"Nesterov Accelerated Gradient","full_name":"Nesterov Accelerated Gradient","full_name_withheld":false,"description_markdown":"**Nesterov Accelerated Gradient** is a momentum-based [SGD](https://paperswithcode.com/method/sgd) optimizer that \"looks ahead\" to where the parameters will be to calculate the gradient **ex post** rather than **ex ante**:\r\n\r\n$$ v\\_{t} = \\gamma{v}\\_{t-1} - \\eta\\nabla\\_{\\theta}J\\left(\\theta_{t-1}+\\gamma{v\\_{t-1}}\\right) $$\r\n$$ \\theta\\_{t} = \\theta\\_{t-1} + v\\_{t} $$\r\n$$ \\gamma, \\eta \\in \\mathbb{R}^+ $$\r\n\r\nLike SGD with momentum $\\gamma$ is usually set to $0.9$. $\\eta$ and $\\gamma$ are usually less than $1$.\r\n\r\nThe intuition is that the [standard momentum](https://paperswithcode.com/method/sgd-with-momentum) method first computes the gradient at the current location and then takes a big jump in the direction of the updated accumulated gradient. In contrast Nesterov momentum first makes a big jump in the direction of the previous accumulated gradient and then measures the gradient where it ends up and makes a correction. The idea being that it is better to correct a mistake after you have made it. \r\n\r\nImage Source: [Geoff Hinton lecture notes](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)","description_state":"present","introduced_year":1983,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Large Batch Optimization","url":"/methods/category/large-batch-optimization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":34,"archive_num_papers":34,"papers_newest_first":[{"paper":null,"title":"Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters","date":"2025-06-13","arxiv_id":"2506.11904","n_code_links":0,"syntology":null},{"paper":"/paper/nesterov-method-for-asynchronous-pipeline","title":"Nesterov Method for Asynchronous Pipeline Parallel Optimization","date":"2025-05-02","arxiv_id":"2505.01099","n_code_links":1,"syntology":null},{"paper":"/paper/advancing-rvfl-networks-robust-classification","title":"Advancing RVFL networks: Robust classification with the HawkEye loss function","date":"2024-10-01","arxiv_id":"2410.00510","n_code_links":1,"syntology":null},{"paper":"/paper/an-accelerated-algorithm-for-stochastic","title":"An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness","date":"2024-09-28","arxiv_id":"2409.19212","n_code_links":1,"syntology":{"ran":3,"of":4,"unverified":1,"pointer_only":0}},{"paper":"/paper/optimizing-time-series-forecasting-a","title":"Optimizing Time Series Forecasting: A Comparative Study of Adam and Nesterov Accelerated Gradient on LSTM and GRU networks Using Stock Market data","date":"2024-09-28","arxiv_id":"2410.01843","n_code_links":1,"syntology":null},{"paper":"/paper/denomamba-a-fused-state-space-model-for-low","title":"DenoMamba: A fused state-space model for low-dose CT denoising","date":"2024-09-19","arxiv_id":"2409.13094","n_code_links":1,"syntology":null},{"paper":null,"title":"3D CBCT Challenge 2024: Improved Cone Beam CT Reconstruction using SwinIR-Based Sinogram and Image Enhancement","date":"2024-06-12","arxiv_id":"2406.08048","n_code_links":0,"syntology":null},{"paper":"/paper/momentum-sam-sharpness-aware-minimization","title":"Momentum-SAM: Sharpness Aware Minimization without Computational Overhead","date":"2024-01-22","arxiv_id":"2401.12033","n_code_links":1,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":2}},{"paper":null,"title":"Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima","date":"2023-07-13","arxiv_id":"2307.07030","n_code_links":0,"syntology":null},{"paper":null,"title":"Riemannian accelerated gradient methods via extrapolation","date":"2022-08-13","arxiv_id":"2208.06619","n_code_links":0,"syntology":null},{"paper":null,"title":"Last-iterate convergence analysis of stochastic momentum methods for neural networks","date":"2022-05-30","arxiv_id":"2205.14811","n_code_links":0,"syntology":null},{"paper":null,"title":"Automated Parking Space Detection Using Convolutional Neural Networks","date":"2021-06-14","arxiv_id":"2106.07228","n_code_links":0,"syntology":null},{"paper":null,"title":"A Discrete Variational Derivation of Accelerated Methods in Optimization","date":"2021-06-04","arxiv_id":"2106.02700","n_code_links":0,"syntology":null},{"paper":"/paper/a-large-batch-optimizer-reality-check","title":"A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes","date":"2021-02-12","arxiv_id":"2102.06356","n_code_links":0,"syntology":null},{"paper":null,"title":"Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance","date":"2020-12-07","arxiv_id":"2012.04002","n_code_links":0,"syntology":null},{"paper":"/paper/a-dynamical-view-on-optimization-algorithms","title":"A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks","date":"2020-10-25","arxiv_id":"2010.13165","n_code_links":1,"syntology":null},{"paper":"/paper/improving-convergence-for-nonconvex-composite","title":"Accelerated Gradient Methods for Sparse Statistical Learning with Nonconvex Penalties","date":"2020-09-22","arxiv_id":"2009.10629","n_code_links":1,"syntology":null},{"paper":null,"title":"Federated Learning with Nesterov Accelerated Gradient","date":"2020-09-18","arxiv_id":"2009.08716","n_code_links":0,"syntology":null},{"paper":"/paper/greedynas-towards-fast-one-shot-nas-with","title":"GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet","date":"2020-03-25","arxiv_id":"2003.11236","n_code_links":0,"syntology":null},{"paper":"/paper/scheduled-restart-momentum-for-accelerated","title":"Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent","date":"2020-02-24","arxiv_id":"2002.10583","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":1}},{"paper":null,"title":"Second-order Information in First-order Optimization Methods","date":"2019-12-20","arxiv_id":"1912.09926","n_code_links":0,"syntology":null},{"paper":"/paper/liteseg-a-novel-lightweight-convnet-for","title":"LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation","date":"2019-12-13","arxiv_id":"1912.06683","n_code_links":2,"syntology":null},{"paper":null,"title":"Best Practices for Convolutional Neural Networks Applied to Object Recognition in Images","date":"2019-10-29","arxiv_id":"1910.13029","n_code_links":0,"syntology":null},{"paper":"/paper/nesterov-accelerated-gradient-and-scale","title":"Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks","date":"2019-08-17","arxiv_id":"1908.06281","n_code_links":3,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}},{"paper":null,"title":"Power Gradient Descent","date":"2019-06-11","arxiv_id":"1906.04787","n_code_links":0,"syntology":null},{"paper":null,"title":"DANA: Scalable Out-of-the-box Distributed ASGD Without Retuning","date":"2019-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/bag-of-tricks-for-image-classification-with","title":"Bag of Tricks for Image Classification with Convolutional Neural Networks","date":"2018-12-04","arxiv_id":"1812.01187","n_code_links":28,"syntology":{"ran":4,"of":15,"unverified":11,"pointer_only":5}},{"paper":"/paper/nesterov-acceleration-of-alternating-least","title":"Nesterov Acceleration of Alternating Least Squares for Canonical Tensor Decomposition: Momentum Step Size Selection and Restart Mechanisms","date":"2018-10-13","arxiv_id":"1810.05846","n_code_links":1,"syntology":null},{"paper":null,"title":"A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration","date":"2018-08-10","arxiv_id":"1808.03408","n_code_links":0,"syntology":null},{"paper":"/paper/big-little-net-an-efficient-multi-scale","title":"Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition","date":"2018-07-10","arxiv_id":"1807.03848","n_code_links":3,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":8},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":5},{"task":"/task/image-classification","name":"image-classification","papers":4},{"task":"/task/classification","name":"General Classification","papers":3},{"task":"/task/object-recognition","name":"Object Recognition","papers":3},{"task":"/task/denoising","name":"Denoising","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":2},{"task":"/task/2d-human-pose-estimation","name":"2D Human Pose Estimation","papers":1},{"task":"/task/adversarial-attack","name":"Adversarial Attack","papers":1},{"task":"/task/all","name":"All","papers":1},{"task":"/task/bilevel-optimization","name":"Bilevel Optimization","papers":1},{"task":"/task/breast-tumour-classification","name":"Breast Tumour Classification","papers":1},{"task":"/task/ct-reconstruction","name":"CT Reconstruction","papers":1},{"task":"/task/classification-1","name":"Classification","papers":1},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":1},{"task":"/task/crowd-counting","name":"Crowd Counting","papers":1},{"task":"/task/decoder","name":"Decoder","papers":1},{"task":"/task/diagnostic","name":"Diagnostic","papers":1},{"task":"/task/distributed-computing","name":"Distributed Computing","papers":1}],"tasks_shown":20,"n_tasks":59,"usage_by_year":[{"year":"2016","papers":4},{"year":"2018","papers":4},{"year":"2019","papers":6},{"year":"2020","papers":6},{"year":"2021","papers":3},{"year":"2022","papers":2},{"year":"2023","papers":1},{"year":"2024","papers":6},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/nesterov-accelerated-gradient"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}