Methods › General › Large Batch Optimization › Nesterov Accelerated Gradient
Nesterov Accelerated Gradient
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Nesterov Accelerated Gradient is a momentum-based SGD optimizer that "looks ahead" to where the parameters will be to calculate the gradient ex post rather than ex ante:
vₜ = γvₜ₋₁ - η∇_θJ(θₜ₋₁+γvₜ₋₁) θₜ = θₜ₋₁ + vₜ γ, η∈ℝ^+
Like SGD with momentum γ is usually set to $0.9$. η and γ are usually less than $1$.
The intuition is that the standard momentum method first computes the gradient at the current location and then takes a big jump in the direction of the updated accumulated gradient. In contrast Nesterov momentum first makes a big jump in the direction of the previous accumulated gradient and then measures the gradient where it ends up and makes a correction. The idea being that it is better to correct a mistake after you have made it.
Image Source: Geoff Hinton lecture notes
Papers archive 2025-07-28
30 shown of 34, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters 13 Jun 2025 · 0 repositories · arXiv:2506.11904
-
Nesterov Method for Asynchronous Pipeline Parallel Optimization 2 May 2025 · 1 repository · arXiv:2505.01099
-
Advancing RVFL networks: Robust classification with the HawkEye loss function 1 Oct 2024 · 1 repository · arXiv:2410.00510
-
An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness 28 Sep 2024 · 1 repository · arXiv:2409.19212Syntology ran 3 of 4 samples · 1 unverified
-
Optimizing Time Series Forecasting: A Comparative Study of Adam and Nesterov Accelerated Gradient on LSTM and GRU networks Using Stock Market data 28 Sep 2024 · 1 repository · arXiv:2410.01843
-
DenoMamba: A fused state-space model for low-dose CT denoising 19 Sep 2024 · 1 repository · arXiv:2409.13094
-
3D CBCT Challenge 2024: Improved Cone Beam CT Reconstruction using SwinIR-Based Sinogram and Image Enhancement 12 Jun 2024 · 0 repositories · arXiv:2406.08048
-
Momentum-SAM: Sharpness Aware Minimization without Computational Overhead 22 Jan 2024 · 1 repository · arXiv:2401.12033Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
-
Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima 13 Jul 2023 · 0 repositories · arXiv:2307.07030
-
Riemannian accelerated gradient methods via extrapolation 13 Aug 2022 · 0 repositories · arXiv:2208.06619
-
Last-iterate convergence analysis of stochastic momentum methods for neural networks 30 May 2022 · 0 repositories · arXiv:2205.14811
-
Automated Parking Space Detection Using Convolutional Neural Networks 14 Jun 2021 · 0 repositories · arXiv:2106.07228
-
A Discrete Variational Derivation of Accelerated Methods in Optimization 4 Jun 2021 · 0 repositories · arXiv:2106.02700
-
A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes 12 Feb 2021 · 0 repositories · arXiv:2102.06356
-
Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance 7 Dec 2020 · 0 repositories · arXiv:2012.04002
-
A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks 25 Oct 2020 · 1 repository · arXiv:2010.13165
-
Accelerated Gradient Methods for Sparse Statistical Learning with Nonconvex Penalties 22 Sep 2020 · 1 repository · arXiv:2009.10629
-
Federated Learning with Nesterov Accelerated Gradient 18 Sep 2020 · 0 repositories · arXiv:2009.08716
-
GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet 25 Mar 2020 · 0 repositories · arXiv:2003.11236
-
Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent 24 Feb 2020 · 1 repository · arXiv:2002.10583Syntology ran 0 of 1 samples · 1 unverified · 1 pointer-only (licence)
-
Second-order Information in First-order Optimization Methods 20 Dec 2019 · 0 repositories · arXiv:1912.09926
-
LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation 13 Dec 2019 · 2 repositories · arXiv:1912.06683
-
Best Practices for Convolutional Neural Networks Applied to Object Recognition in Images 29 Oct 2019 · 0 repositories · arXiv:1910.13029
-
Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks 17 Aug 2019 · 3 repositories · arXiv:1908.06281Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
-
Power Gradient Descent 11 Jun 2019 · 0 repositories · arXiv:1906.04787
-
DANA: Scalable Out-of-the-box Distributed ASGD Without Retuning 1 May 2019 · 0 repositories
-
Bag of Tricks for Image Classification with Convolutional Neural Networks 4 Dec 2018 · 28 repositories · arXiv:1812.01187Syntology ran 4 of 15 samples · 11 unverified · 5 pointer-only (licence)
-
Nesterov Acceleration of Alternating Least Squares for Canonical Tensor Decomposition: Momentum Step Size Selection and Restart Mechanisms 13 Oct 2018 · 1 repository · arXiv:1810.05846
-
A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration 10 Aug 2018 · 0 repositories · arXiv:1808.03408
-
Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition 10 Jul 2018 · 3 repositories · arXiv:1807.03848
Tasks archive 2025-07-28
20 shown of 59 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections