{"url":"/method/adagrad","slug":"adagrad","name":"AdaGrad","full_name":"AdaGrad","full_name_withheld":false,"description_markdown":"**AdaGrad** is a stochastic optimization method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring features, and larger updates for parameters associated with infrequently occurring features. In its update rule, Adagrad modifies the general learning rate $\\eta$ at each time step $t$ for every parameter $\\theta\\_{i}$ based on the past gradients for $\\theta\\_{i}$: \r\n\r\n$$ \\theta\\_{t+1, i} = \\theta\\_{t, i} - \\frac{\\eta}{\\sqrt{G\\_{t, ii} + \\epsilon}}g\\_{t, i} $$\r\n\r\nThe benefit of AdaGrad is that it eliminates the need to manually tune the learning rate; most leave it at a default value of $0.01$. Its main weakness is the accumulation of the squared gradients in the denominator. Since every added term is positive, the accumulated sum keeps growing during training, causing the learning rate to shrink and becoming infinitesimally small.\r\n\r\nImage: [Alec Radford](https://twitter.com/alecrad)","description_state":"present","introduced_year":2011,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":"https://github.com/Dawn-Of-Eve/nadir/blob/main/src/nadir/adagrad.py","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Large Batch Optimization","url":"/methods/category/large-batch-optimization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":191,"archive_num_papers":192,"papers_newest_first":[{"paper":null,"title":"Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization","date":"2025-07-15","arxiv_id":"2507.11513","n_code_links":0,"syntology":null},{"paper":null,"title":"LightSAM: Parameter-Agnostic Sharpness-Aware Minimization","date":"2025-05-30","arxiv_id":"2505.24399","n_code_links":0,"syntology":null},{"paper":"/paper/sample-and-computationally-efficient-1","title":"Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation","date":"2025-05-20","arxiv_id":"2505.14821","n_code_links":1,"syntology":null},{"paper":null,"title":"Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness","date":"2025-05-07","arxiv_id":"2505.04599","n_code_links":0,"syntology":null},{"paper":null,"title":"Structured Preconditioners in Adaptive Optimization: A Unified Analysis","date":"2025-03-13","arxiv_id":"2503.10537","n_code_links":0,"syntology":null},{"paper":null,"title":"Tractable Representations for Convergent Approximation of Distributional HJB Equations","date":"2025-03-07","arxiv_id":"2503.05563","n_code_links":0,"syntology":null},{"paper":null,"title":"Symmetric Rank-One Quasi-Newton Methods for Deep Learning Using Cubic Regularization","date":"2025-02-17","arxiv_id":"2502.12298","n_code_links":0,"syntology":null},{"paper":null,"title":"Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions","date":"2025-01-08","arxiv_id":"2501.04437","n_code_links":0,"syntology":null},{"paper":null,"title":"Towards Simple and Provable Parameter-Free Adaptive Gradient Methods","date":"2024-12-27","arxiv_id":"2412.19444","n_code_links":0,"syntology":null},{"paper":null,"title":"Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training","date":"2024-12-06","arxiv_id":"2412.04718","n_code_links":0,"syntology":null},{"paper":null,"title":"A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation","date":"2024-11-19","arxiv_id":"2411.12157","n_code_links":0,"syntology":null},{"paper":"/paper/modeling-adagrad-rmsprop-and-adam-with","title":"Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations","date":"2024-11-14","arxiv_id":"2411.09734","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":null,"title":"New Insight in Cervical Cancer Diagnosis Using Convolution Neural Network Architecture","date":"2024-10-23","arxiv_id":"2410.17735","n_code_links":0,"syntology":null},{"paper":null,"title":"Preconditioning for Accelerated Gradient Descent Optimization and Regularization","date":"2024-09-30","arxiv_id":"2410.00232","n_code_links":0,"syntology":null},{"paper":null,"title":"Stability and convergence analysis of AdaGrad for non-convex optimization via novel stopping time-based techniques","date":"2024-09-08","arxiv_id":"2409.05023","n_code_links":0,"syntology":null},{"paper":"/paper/causal-temporal-representation-learning-with","title":"Causal Temporal Representation Learning with Nonstationary Sparse Transition","date":"2024-09-05","arxiv_id":"2409.03142","n_code_links":1,"syntology":{"ran":7,"of":9,"unverified":2,"pointer_only":9}},{"paper":"/paper/machine-learning-models-for-daily-rainfall","title":"Machine learning models for daily rainfall forecasting in Northern Tropical Africa using tropical wave predictors","date":"2024-08-29","arxiv_id":"2408.16349","n_code_links":1,"syntology":null},{"paper":null,"title":"A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality","date":"2024-07-17","arxiv_id":"2407.12629","n_code_links":0,"syntology":null},{"paper":null,"title":"AdaGrad under Anisotropic Smoothness","date":"2024-06-21","arxiv_id":"2406.15244","n_code_links":0,"syntology":null},{"paper":null,"title":"Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization","date":"2024-06-07","arxiv_id":"2406.04592","n_code_links":0,"syntology":null},{"paper":"/paper/gradient-clipping-improves-adagrad-when-the","title":"Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed","date":"2024-06-06","arxiv_id":"2406.04443","n_code_links":1,"syntology":null},{"paper":null,"title":"Tight Bounds for Online Convex Optimization with Adversarial Constraints","date":"2024-05-15","arxiv_id":"2405.09296","n_code_links":0,"syntology":null},{"paper":null,"title":"CTRL: Continuous-Time Representation Learning on Temporal Heterogeneous Information Network","date":"2024-05-11","arxiv_id":"2405.08013","n_code_links":0,"syntology":null},{"paper":"/paper/classp-a-biologically-inspired-approach-to-1","title":"CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion","date":"2024-04-29","arxiv_id":"2405.09637","n_code_links":1,"syntology":null},{"paper":null,"title":"Deep Learning with Parametric Lenses","date":"2024-03-30","arxiv_id":"2404.00408","n_code_links":0,"syntology":null},{"paper":"/paper/an-embarrassingly-simple-defense-against","title":"Towards Adversarial Robustness And Backdoor Mitigation in SSL","date":"2024-03-23","arxiv_id":"2403.15918","n_code_links":1,"syntology":null},{"paper":null,"title":"Adaptive Federated Learning Over the Air","date":"2024-03-11","arxiv_id":"2403.06528","n_code_links":0,"syntology":null},{"paper":"/paper/remove-that-square-root-a-new-efficient-scale","title":"Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad","date":"2024-03-05","arxiv_id":"2403.02648","n_code_links":1,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":2}},{"paper":null,"title":"Revisiting Convergence of AdaGrad with Relaxed Assumptions","date":"2024-02-21","arxiv_id":"2402.13794","n_code_links":0,"syntology":null},{"paper":"/paper/stochastic-hessian-fitting-on-lie-group","title":"Stochastic Hessian Fittings with Lie Groups","date":"2024-02-19","arxiv_id":"2402.11858","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":26},{"task":"/task/language-modelling","name":"Language Modelling","papers":14},{"task":"/task/language-modeling","name":"Language Modeling","papers":13},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":7},{"task":"/task/image-classification","name":"Image Classification","papers":7},{"task":"/task/representation-learning","name":"Representation Learning","papers":7},{"task":"/task/second-order-methods","name":"Second-order methods","papers":7},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":6},{"task":"/task/image-classification","name":"image-classification","papers":6},{"task":"/task/text-generation","name":"Text Generation","papers":5},{"task":"/task/translation","name":"Translation","papers":4},{"task":"/task/deep-learning","name":"Deep Learning","papers":3},{"task":"/task/distributed-optimization","name":"Distributed Optimization","papers":3},{"task":"/task/federated-learning","name":"Federated Learning","papers":3},{"task":null,"name":"GPU","papers":3},{"task":"/task/link-prediction","name":"Link Prediction","papers":3},{"task":"/task/machine-translation","name":"Machine Translation","papers":3},{"task":"/task/multi-task-learning","name":"Multi-Task Learning","papers":3},{"task":"/task/prediction","name":"Prediction","papers":3},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":3}],"tasks_shown":20,"n_tasks":118,"usage_by_year":[{"year":"2013","papers":1},{"year":"2015","papers":4},{"year":"2016","papers":4},{"year":"2017","papers":8},{"year":"2018","papers":12},{"year":"2019","papers":20},{"year":"2020","papers":22},{"year":"2021","papers":29},{"year":"2022","papers":30},{"year":"2023","papers":25},{"year":"2024","papers":28},{"year":"2025","papers":8}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adagrad"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}