Papers › SGD momentum optimizer with step estimation by online parabola model
SGD momentum optimizer with step estimation by online parabola model
Jarek Duda
In stochastic gradient descent, especially for neural network training, there are currently dominating first order methods: not modeling local distance to minimum. This information required for optimal step size is provided by second order methods, however, they have many difficulties, starting with full Hessian having square of dimension number of coefficients. This article proposes a minimal step from successful first order momentum method toward second order: online parabola modelling in just a single direction: normalized v̂ from momentum method. It is done by estimating linear trend of gradients g⃗=∇F(θ⃗) in v̂ direction: such that $g(\vec{\theta}_\bot+\theta\hat{v})\approx \lambda (\theta -p)$ for θ= θ⃗·v̂, g= g⃗·v̂, $\vec{\theta}_\bot=\vec{\theta}-\theta\hat{v}$. Using linear regression, λ, p are MSE estimated by just updating four averages (of g, θ, gθ, θ²) in the considered direction. Exponential moving averages allow here for inexpensive online estimation, weakening contribution of the old gradients. Controlling sign of curvature λ, we can repel from saddles in contrast to attraction in standard Newton method. In the remaining directions: not considered in second order model, we can simultaneously perform e.g. gradient descent. There is also discussed its learning rate approximation as μ=σ_θ / σ_g, allowing e.g. for adaptive SGD - with learning rate separately optimized (2nd order) for each parameter.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections