{"url":"/method/amsbound","slug":"amsbound","name":"AMSBound","full_name":"AMSBound","full_name_withheld":false,"description_markdown":"**AMSBound** is a variant of the [AMSGrad](https://paperswithcode.com/method/amsgrad) stochastic optimizer which is designed to be more robust to extreme learning rates. Dynamic bounds are employed on learning rates, where the lower and upper bound are initialized as zero and infinity respectively, and they both smoothly converge to a constant final step size. AMSBound can be regarded as an adaptive method at the beginning of training, and it gradually and smoothly transforms to [SGD](https://paperswithcode.com/method/sgd) (or with momentum) as time step increases. \r\n\r\n$$ g\\_{t} = \\nabla{f}\\_{t}\\left(x\\_{t}\\right) $$\r\n\r\n$$ m\\_{t} = \\beta\\_{1t}m\\_{t-1} + \\left(1-\\beta\\_{1t}\\right)g\\_{t} $$\r\n\r\n$$ v\\_{t} = \\beta\\_{2}v\\_{t-1} + \\left(1-\\beta\\_{2}\\right)g\\_{t}^{2}$$\r\n\r\n$$ \\hat{v}\\_{t} = \\max\\left(\\hat{v}\\_{t-1}, v\\_{t}\\right) \\text{ and } V\\_{t} = \\text{diag}\\left(\\hat{v}\\_{t}\\right) $$\r\n\r\n$$ \\eta = \\text{Clip}\\left(\\alpha/\\sqrt{V\\_{t}}, \\eta\\_{l}\\left(t\\right), \\eta\\_{u}\\left(t\\right)\\right) \\text{ and } \\eta\\_{t} = \\eta/\\sqrt{t} $$\r\n\r\n$$ x\\_{t+1} = \\Pi\\_{\\mathcal{F}, \\text{diag}\\left(\\eta\\_{t}^{-1}\\right)}\\left(x\\_{t} - \\eta\\_{t} \\odot m\\_{t} \\right) $$\r\n\r\nWhere $\\alpha$ is the initial step size, and $\\eta_{l}$ and $\\eta_{u}$ are the lower and upper bound functions respectively.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Adaptive Gradient Methods with Dynamic Bound of Learning Rate","paper":"/paper/adaptive-gradient-methods-with-dynamic-bound","first_author":"Liangchen Luo","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/adaptive-gradient-methods-with-dynamic-bound"},"source":{"url":"http://arxiv.org/abs/1902.09843v1","title":"Adaptive Gradient Methods with Dynamic Bound of Learning Rate","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/jettify/pytorch-optimizer/blob/155246597d66dd774156599be0f07a8c6f7758aa/torch_optimizer/adabound.py#L11","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":"/paper/adaptive-gradient-methods-with-dynamic-bound","title":"Adaptive Gradient Methods with Dynamic Bound of Learning Rate","date":"2019-02-26","arxiv_id":"1902.09843","n_code_links":5,"syntology":null}],"papers_shown":1,"tasks":[],"tasks_shown":0,"n_tasks":0,"usage_by_year":[{"year":"2019","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/amsbound"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}