{"url":"/method/adasmooth","slug":"adasmooth","name":"AdaSmooth","full_name":"Adaptive Smooth Optimizer","full_name_withheld":false,"description_markdown":"**AdaSmooth** is a stochastic optimization technique that allows for per-dimension learning rate method for [SGD](https://paperswithcode.com/method/sgd). It is an extension of [Adagrad](https://paperswithcode.com/method/adagrad) and [AdaDelta](https://paperswithcode.com/method/adadelta) that seek to reduce its aggressive, monotonically decreasing learning rate. Instead of accumulating all past squared gradients, Adadelta restricts the window of accumulated past gradients to a fixed size $w$ while AdaSmooth adaptively selects the size of the window.\r\n\r\nGiven the window size  $M$, the effective ratio is calculated by \r\n\r\n$$e_t  = \\frac{s_t}{n_t}= \\frac{| x_t -  x_{t-M}|}{\\sum_{i=0}^{M-1} | x_{t-i} -  x_{t-1-i}|}\\\\\r\n= \\frac{| \\sum_{i=0}^{M-1} \\Delta x_{t-1-i}|}{\\sum_{i=0}^{M-1} | \\Delta x_{t-1-i}|}.$$\r\n\r\nGiven the effective ratio, the scaled smoothing constant is obtained by:\r\n\r\n$$c_t =  ( \\rho_2- \\rho_1) \\times e_t   + (1-\\rho_2),$$\r\n\r\nThe running average $E\\left[g^{2}\\right]\\_{t}$ at time step $t$ then depends only on the previous average and current gradient:\r\n\r\n$$ E\\left[g^{2}\\right]\\_{t} = c_t^2 \\odot g_{t}^2  +  \\left(1-c_t^2 \\right)\\odot E[g^2]_{t-1} $$\r\n\r\nUsually $\\rho_1$ is set to around $0.5$ and $\\rho_2$ is set to around 0.99. The update step the follows:\r\n\r\n$$ \\Delta x_t = -\\frac{\\eta}{\\sqrt{E\\left[g^{2}\\right]\\_{t} + \\epsilon}} \\odot  g_{t}, $$\r\n\r\nwhich is incorporated into the final update:\r\n\r\n$$x_{t+1} = x_{t} + \\Delta x_t.$$\r\n\r\nThe main advantage of AdaSmooth is its faster convergence rate and insensitivity to hyperparameters.","description_state":"present","introduced_year":null,"introduced_by":{"title":"AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio","paper":"/paper/adasmooth-an-adaptive-learning-rate-method","first_author":"Jun Lu","n_authors":1,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/adasmooth-an-adaptive-learning-rate-method"},"source":{"url":"https://arxiv.org/abs/2204.00825v1","title":"AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":null,"title":"Gradient Descent, Stochastic Optimization, and Other Tales","date":"2022-05-02","arxiv_id":"2205.00832","n_code_links":0,"syntology":null},{"paper":"/paper/adasmooth-an-adaptive-learning-rate-method","title":"AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio","date":"2022-04-02","arxiv_id":"2204.00825","n_code_links":0,"syntology":null}],"papers_shown":2,"tasks":[{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":2}],"tasks_shown":1,"n_tasks":1,"usage_by_year":[{"year":"2022","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adasmooth"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}