Papers › Backtracking gradient descent method for general C¹ functions, with applications to...
Backtracking gradient descent method for general C¹ functions, with applications to Deep Learning
Tuyen Trung Truong, Tuan Hang Nguyen
While Standard gradient descent is one very popular optimisation method, its convergence cannot be proven beyond the class of functions whose gradient is globally Lipschitz continuous. As such, it is not actually applicable to realistic applications such as Deep Neural Networks. In this paper, we prove that its backtracking variant behaves very nicely, in particular convergence can be shown for all Morse functions. The main theoretical result of this paper is as follows. Theorem. Let f:ℝᵏ→ℝ be a C¹ function, and {zₙ} a sequence constructed from the Backtracking gradient descent algorithm. (1) Either lim_(n→∞)||zₙ||=∞ or lim_(n→∞)||zₙ₊₁-zₙ||=0. (2) Assume that f has at most countably many critical points. Then either lim_(n→∞)||zₙ||=∞ or {zₙ} converges to a critical point of f. (3) More generally, assume that all connected components of the set of critical points of f are compact. Then either lim_(n→∞)||zₙ||=∞ or {zₙ} is bounded. Moreover, in the latter case the set of cluster points of {zₙ} is connected. Some generalised versions of this result, including an inexact version, are included. Another result in this paper concerns the problem of saddle points. We then present a heuristic argument to explain why Standard gradient descent method works so well, and modifications of the backtracking versions of GD, MMT and NAG. Experiments with datasets CIFAR10 and CIFAR100 on various popular architectures verify the heuristic argument also for the mini-batch practice and show that our new algorithms, while automatically fine tuning learning rates, perform better than current state-of-the-art methods such as MMT, NAG, Adagrad, Adadelta, RMSProp, Adam and Adamax.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections