Methods › General › Large Batch Optimization › AdaGrad
AdaGrad
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AdaGrad is a stochastic optimization method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring features, and larger updates for parameters associated with infrequently occurring features. In its update rule, Adagrad modifies the general learning rate η at each time step t for every parameter θᵢ based on the past gradients for θᵢ:
θ_(t+1, i) = θ_(t, i) - η/(√(G_(t, ii) + ϵ))g_(t, i)
The benefit of AdaGrad is that it eliminates the need to manually tune the learning rate; most leave it at a default value of 0.01. Its main weakness is the accumulation of the squared gradients in the denominator. Since every added term is positive, the accumulated sum keeps growing during training, causing the learning rate to shrink and becoming infinitesimally small.
Image: Alec Radford
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: AdaGrad-Diff; AdaGrad variant; Piecewise AdaGrad.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 191 · with a code link: 70 · where Syntology ran a sample: 23 (19 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (23 of 191 tagged: 19 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 191, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization 15 Jul 2025 · 0 repositories · arXiv:2507.11513
-
LightSAM: Parameter-Agnostic Sharpness-Aware Minimization 30 May 2025 · 0 repositories · arXiv:2505.24399
-
Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation 20 May 2025 · 1 repository · arXiv:2505.14821
-
Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness 7 May 2025 · 0 repositories · arXiv:2505.04599
-
Structured Preconditioners in Adaptive Optimization: A Unified Analysis 13 Mar 2025 · 0 repositories · arXiv:2503.10537
-
Tractable Representations for Convergent Approximation of Distributional HJB Equations 7 Mar 2025 · 0 repositories · arXiv:2503.05563
-
Symmetric Rank-One Quasi-Newton Methods for Deep Learning Using Cubic Regularization 17 Feb 2025 · 0 repositories · arXiv:2502.12298
-
Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions 8 Jan 2025 · 0 repositories · arXiv:2501.04437
-
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods 27 Dec 2024 · 0 repositories · arXiv:2412.19444
-
Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training 6 Dec 2024 · 0 repositories · arXiv:2412.04718
-
A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation 19 Nov 2024 · 0 repositories · arXiv:2411.12157
-
Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations 14 Nov 2024 · 1 repository · arXiv:2411.09734Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
New Insight in Cervical Cancer Diagnosis Using Convolution Neural Network Architecture 23 Oct 2024 · 0 repositories · arXiv:2410.17735
-
Preconditioning for Accelerated Gradient Descent Optimization and Regularization 30 Sep 2024 · 0 repositories · arXiv:2410.00232
-
Stability and convergence analysis of AdaGrad for non-convex optimization via novel stopping time-based techniques 8 Sep 2024 · 0 repositories · arXiv:2409.05023
-
Causal Temporal Representation Learning with Nonstationary Sparse Transition 5 Sep 2024 · 1 repository · arXiv:2409.03142Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples) · 9 pointer-only (licence)
-
Machine learning models for daily rainfall forecasting in Northern Tropical Africa using tropical wave predictors 29 Aug 2024 · 1 repository · arXiv:2408.16349
-
A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality 17 Jul 2024 · 0 repositories · arXiv:2407.12629
-
AdaGrad under Anisotropic Smoothness 21 Jun 2024 · 0 repositories · arXiv:2406.15244
-
Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization 7 Jun 2024 · 0 repositories · arXiv:2406.04592
-
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed 6 Jun 2024 · 1 repository · arXiv:2406.04443
-
Tight Bounds for Online Convex Optimization with Adversarial Constraints 15 May 2024 · 0 repositories · arXiv:2405.09296
-
CTRL: Continuous-Time Representation Learning on Temporal Heterogeneous Information Network 11 May 2024 · 0 repositories · arXiv:2405.08013
-
CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion 29 Apr 2024 · 1 repository · arXiv:2405.09637
-
Deep Learning with Parametric Lenses 30 Mar 2024 · 0 repositories · arXiv:2404.00408
-
Towards Adversarial Robustness And Backdoor Mitigation in SSL 23 Mar 2024 · 1 repository · arXiv:2403.15918
-
Adaptive Federated Learning Over the Air 11 Mar 2024 · 0 repositories · arXiv:2403.06528
-
Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad 5 Mar 2024 · 1 repository · arXiv:2403.02648Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Revisiting Convergence of AdaGrad with Relaxed Assumptions 21 Feb 2024 · 0 repositories · arXiv:2402.13794
-
Stochastic Hessian Fittings with Lie Groups 19 Feb 2024 · 1 repository · arXiv:2402.11858
-
AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods 17 Feb 2024 · 0 repositories · arXiv:2402.11215
-
Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks 7 Feb 2024 · 0 repositories · arXiv:2402.05155
-
Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matching 7 Feb 2024 · 1 repository · arXiv:2402.04924Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 1 pointer-only (licence)
-
Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation 5 Feb 2024 · 1 repository · arXiv:2402.02857
-
Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence 2 Feb 2024 · 0 repositories · arXiv:2402.01515
-
HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy 26 Jan 2024 · 1 repository · arXiv:2401.15207Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms 28 Dec 2023 · 1 repository · arXiv:2312.17369Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 10 harvested samples) · 10 pointer-only (licence)
-
Optimal Algorithms for Online Convex Optimization with Adversarial Constraints 29 Oct 2023 · 0 repositories · arXiv:2310.18955
-
Towards Hyperparameter-Agnostic DNN Training via Dynamical System Insights 21 Oct 2023 · 0 repositories · arXiv:2310.13901
-
A Parallelized, Adam-Based Solver for Reserve and Security Constrained AC Unit Commitment 10 Oct 2023 · 1 repository · arXiv:2310.06650
-
Stochastic Gradient Descent with Preconditioned Polyak Step-size 3 Oct 2023 · 1 repository · arXiv:2310.02093Syntology official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 15 harvested samples) · 15 pointer-only (licence)
-
A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale 12 Sep 2023 · 3 repositories · arXiv:2309.06497Syntology official: harvested, nothing ran · 0 ran · 2 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Adaptively Optimised Adaptive Importance Samplers 18 Jul 2023 · 0 repositories · arXiv:2307.09341
-
Error Feedback Can Accurately Compress Preconditioners 9 Jun 2023 · 1 repository · arXiv:2306.06098Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
Prodigy: An Expeditiously Adaptive Parameter-Free Learner 9 Jun 2023 · 1 repository · arXiv:2306.06101
-
CTRL: Connect Collaborative and Language Model for CTR Prediction 5 Jun 2023 · 0 repositories · arXiv:2306.02841
-
Parameter-free projected gradient descent 31 May 2023 · 0 repositories · arXiv:2305.19605
-
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions 29 May 2023 · 0 repositories · arXiv:2305.18471
-
DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method 25 May 2023 · 3 repositories · arXiv:2305.16284
-
NODDLE: Node2vec based deep learning model for link prediction 25 May 2023 · 0 repositories · arXiv:2305.16421
-
Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods 21 May 2023 · 0 repositories · arXiv:2305.12475
-
Stochastic Ratios Tracking Algorithm for Large Scale Machine Learning Problems 17 May 2023 · 0 repositories · arXiv:2305.09978
-
Optimized Machine Learning for CHD Detection using 3D CNN-based Segmentation, Transfer Learning and Adagrad Optimization 30 Apr 2023 · 0 repositories · arXiv:2305.00411
-
SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish 27 Apr 2023 · 1 repository · arXiv:2304.13994
-
Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection 24 Apr 2023 · 2 repositories · arXiv:2304.12315Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 5 harvested samples)
-
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation 30 Mar 2023 · 1 repository · arXiv:2303.17574
-
Mixing Backward- with Forward-Chaining for Metacognitive Skill Acquisition and Transfer 18 Mar 2023 · 0 repositories · arXiv:2303.12223
-
Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications 1 Mar 2023 · 0 repositories · arXiv:2303.01370
-
High Probability Convergence of Stochastic Gradient Methods 28 Feb 2023 · 0 repositories · arXiv:2302.14843
-
SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance 17 Feb 2023 · 0 repositories · arXiv:2302.08783
-
Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training 14 Feb 2023 · 0 repositories · arXiv:2302.07049
-
Automatically Bounding the Taylor Remainder Series: Tighter Bounds and New Applications 22 Dec 2022 · 1 repository · arXiv:2212.11429
-
AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning 28 Nov 2022 · 0 repositories · arXiv:2211.15055
-
Alternative formulations for gilthead seabream diets: towards a more sustainable production 3 Nov 2022 · 0 repositories · arXiv:2211.02430
-
Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model 31 Oct 2022 · 1 repository · arXiv:2210.17418
-
Unsupervised Learning of Structured Representations via Closed-Loop Transcription 30 Oct 2022 · 1 repository · arXiv:2210.16782
-
An Embarrassingly Simple Backdoor Attack on Self-supervised Learning 13 Oct 2022 · 4 repositories · arXiv:2210.07346Syntology community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Divergence Results and Convergence of a Variance Reduced Version of ADAM 11 Oct 2022 · 0 repositories · arXiv:2210.05607
-
On the Convergence of AdaGrad(Norm) on ℝᵈ: Beyond Convexity, Non-Asymptotic Rate and Acceleration 29 Sep 2022 · 0 repositories · arXiv:2209.14827
-
Continual Learning, Fast and Slow 6 Sep 2022 · 1 repository · arXiv:2209.02370
-
Dynamic Regret of Adaptive Gradient Methods for Strongly Convex Problems 4 Sep 2022 · 0 repositories · arXiv:2209.01608
-
CTRL: Clustering Training Losses for Label Error Detection 17 Aug 2022 · 1 repository · arXiv:2208.08464
-
Multinomial Logistic Regression Algorithms via Quadratic Gradient 14 Aug 2022 · 1 repository · arXiv:2208.06828
-
Quantized Adaptive Subgradient Algorithms and Their Applications 11 Aug 2022 · 0 repositories · arXiv:2208.05631
-
Formal guarantees for heuristic optimization algorithms used in machine learning 31 Jul 2022 · 0 repositories · arXiv:2208.00502
-
Accelerated Federated Learning with Decoupled Adaptive Optimization 14 Jul 2022 · 0 repositories · arXiv:2207.07223
-
BFE and AdaBFE: A New Approach in Learning Rate Automation for Stochastic Optimization 6 Jul 2022 · 0 repositories · arXiv:2207.02763
-
Pursuit of a Discriminative Representation for Multiple Subspaces via Sequential Games 18 Jun 2022 · 1 repository · arXiv:2206.09120Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method 14 Jun 2022 · 0 repositories · arXiv:2206.06900
-
A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning 4 Jun 2022 · 0 repositories · arXiv:2206.02034
-
Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization 1 Jun 2022 · 0 repositories · arXiv:2206.00743
-
Incorporating the Barzilai-Borwein Adaptive Step Size into Sugradient Methods for Deep Network Training 27 May 2022 · 0 repositories · arXiv:2205.13711
-
Symmetry Teleportation for Accelerated Optimization 21 May 2022 · 1 repository · arXiv:2205.10637Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
Efficient and Training-Free Control of Language Generation 12 May 2022 · 0 repositories · arXiv:2205.06036
-
High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize 6 Apr 2022 · 0 repositories · arXiv:2204.02833
-
AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio 2 Apr 2022 · 0 repositories · arXiv:2204.00825
-
Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Data 29 Mar 2022 · 0 repositories · arXiv:2203.15797
-
A Mini-Block Fisher Method for Deep Neural Networks 8 Feb 2022 · 0 repositories · arXiv:2202.04124
-
Step-size Adaptation Using Exponentiated Gradient Updates 31 Jan 2022 · 0 repositories · arXiv:2202.00145
-
On the Convergence of mSGD and AdaGrad for Stochastic Optimization 26 Jan 2022 · 0 repositories · arXiv:2201.11204
-
Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates 5 Jan 2022 · 1 repository · arXiv:2201.01652
-
Adaptively Customizing Activation Functions for Various Layers 17 Dec 2021 · 1 repository · arXiv:2112.09442
-
A Comparative Study of Transformers on Word Sense Disambiguation 30 Nov 2021 · 0 repositories · arXiv:2111.15417
-
Adaptive Differentially Private Empirical Risk Minimization 14 Oct 2021 · 0 repositories · arXiv:2110.07435
-
KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey 5 Oct 2021 · 0 repositories · arXiv:2110.01858
-
Dynamic Regret Analysis for Online Meta-Learning 29 Sep 2021 · 0 repositories · arXiv:2109.14375
-
L-SR1 Adaptive Regularization by Cubics for Deep Learning 29 Sep 2021 · 0 repositories
-
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation 27 Sep 2021 · 3 repositories · arXiv:2109.13296Syntology 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)
-
On the Convergence of Decentralized Adaptive Gradient Methods 7 Sep 2021 · 0 repositories · arXiv:2109.03194
-
Tom: Leveraging trend of the observed gradients for faster convergence 7 Sep 2021 · 2 repositories · arXiv:2109.03820
Tasks archive 2025-07-28
20 shown of 118 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections