{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptive-gradient-methods-with-dynamic-bound","title":"Adaptive Gradient Methods with Dynamic Bound of Learning Rate","arxiv_id":"1902.09843","date":"2019-02-26","proceeding":"ICLR 2019 5","authors":["Liangchen Luo","Yuanhao Xiong","Yan Liu","Xu sun"],"abstract":"Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been\nproposed to achieve a rapid training process with an element-wise scaling term\non learning rates. Though prevailing, they are observed to generalize poorly\ncompared with SGD or even fail to converge due to unstable and extreme learning\nrates. Recent work has put forward some algorithms such as AMSGrad to tackle\nthis issue but they failed to achieve considerable improvement over existing\nmethods. In our paper, we demonstrate that extreme learning rates can lead to\npoor performance. We provide new variants of Adam and AMSGrad, called AdaBound\nand AMSBound respectively, which employ dynamic bounds on learning rates to\nachieve a gradual and smooth transition from adaptive methods to SGD and give a\ntheoretical proof of convergence. We further conduct experiments on various\npopular tasks and models, which is often insufficient in previous work.\nExperimental results show that new variants can eliminate the generalization\ngap between adaptive methods and SGD and maintain higher learning speed early\nin training at the same time. Moreover, they can bring significant improvement\nover their prototypes, especially on complex deep networks. The implementation\nof the algorithm can be found at https://github.com/Luolc/AdaBound .","url_abs":"http://arxiv.org/abs/1902.09843v1","url_pdf":"http://arxiv.org/pdf/1902.09843v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adaptive-gradient-methods-with-dynamic-bound","repo_url":"https://github.com/Luolc/AdaBound","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"adaptive-gradient-methods-with-dynamic-bound","repo_url":"https://github.com/201419/Optimizer-PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"adaptive-gradient-methods-with-dynamic-bound","repo_url":"https://github.com/wri/restoration-mapper","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"adaptive-gradient-methods-with-dynamic-bound","repo_url":"https://github.com/wri/sentinel-tree-cover","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"adaptive-gradient-methods-with-dynamic-bound","repo_url":"https://github.com/taki0112/AdaBound-Tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"amsbound","method_name":"AMSBound"},{"method_slug":"adabound","method_name":"AdaBound"},{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[{"slug":"amsbound","name":"AMSBound","full_name":"AMSBound"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1902.09843","atlas_url":"https://app.syntology.ai/?focus=1902.09843","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}