Methods › General › Stochastic Optimization › AdaBound
AdaBound
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AdaBound is a variant of the Adam stochastic optimizer which is designed to be more robust to extreme learning rates. Dynamic bounds are employed on learning rates, where the lower and upper bound are initialized as zero and infinity respectively, and they both smoothly converge to a constant final step size. AdaBound can be regarded as an adaptive method at the beginning of training, and thereafter it gradually and smoothly transforms to SGD (or with momentum) as the time step increases.
gₜ = ∇fₜ(xₜ)
mₜ = β₁ₜmₜ₋₁ + (1-β₁ₜ)gₜ
vₜ = β₂vₜ₋₁ + (1-β₂)gₜ² and Vₜ = diag(vₜ)
η̂ₜ = Clip(α/√(Vₜ), ηₗ(t), ηᵤ(t)) and ηₜ = η̂ₜ/√(t)
xₜ₊₁ = Π_(ℱ, diag(ηₜ⁻¹))(xₜ - ηₜ ⊙mₜ )
Where α is the initial step size, and ηₗ and ηᵤ are the lower and upper bound functions respectively.
Papers archive 2025-07-28
11 shown of 11, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization 9 May 2023 · 0 repositories · arXiv:2305.05675
-
Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator 30 Apr 2021 · 0 repositories · arXiv:2104.14840
-
BERT-based Chinese Text Classification for Emergency Domain with a Novel Loss Function 9 Apr 2021 · 0 repositories · arXiv:2104.04197
-
Towards Better Generalization of Adaptive Gradient Methods 1 Dec 2020 · 0 repositories
-
FCM-RDpA: TSK Fuzzy Regression Model Construction Using Fuzzy C-Means Clustering, Regularization, DropRule, and Powerball AdaBelief 30 Nov 2020 · 2 repositories · arXiv:2012.00060
-
MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients 21 Jun 2020 · 1 repository · arXiv:2006.11918
-
MBGD-RDA Training and Rule Pruning for Concise TSK Fuzzy Regression Models 1 Mar 2020 · 0 repositories · arXiv:2003.00608
-
An Adaptive and Momental Bound Method for Stochastic Learning 27 Oct 2019 · 2 repositories · arXiv:1910.12249Syntology ran 0 of 2 samples · 2 unverified
-
On the Convergence of AdaBound and its Connection to SGD 13 Aug 2019 · 2 repositories · arXiv:1908.04457
-
Optimize TSK Fuzzy Systems for Regression Problems: Mini-Batch Gradient Descent with Regularization, DropRule and AdaBound (MBGD-RDA) 26 Mar 2019 · 1 repository · arXiv:1903.10951
-
Adaptive Gradient Methods with Dynamic Bound of Learning Rate 26 Feb 2019 · 5 repositories · arXiv:1902.09843
Tasks archive 2025-07-28
15 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections