{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimal-adaptive-and-accelerated-stochastic","title":"Optimal Adaptive and Accelerated Stochastic Gradient Descent","arxiv_id":"1810.00553","date":"2018-10-01","proceeding":null,"authors":["Qi Deng","Yi Cheng","Guanghui Lan"],"abstract":"Stochastic gradient descent (\\textsc{Sgd}) methods are the most powerful\noptimization tools in training machine learning and deep learning models.\nMoreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a.\nadaptive gradient) methods are the two main techniques to improve the slow\nconvergence of \\textsc{Sgd}. While empirical studies have demonstrated\npotential advantages of combining these two techniques, it remains unknown\nwhether these methods can achieve the optimal rate of convergence for\nstochastic optimization. In this paper, we present a new class of adaptive and\naccelerated stochastic gradient descent methods and show that they exhibit the\noptimal sampling and iteration complexity for stochastic optimization. More\nspecifically, we show that diagonal scaling, initially designed to improve\nvanilla stochastic gradient, can be incorporated into accelerated stochastic\ngradient descent to achieve the optimal rate of convergence for smooth\nstochastic optimization. We also show that momentum, apart from being known to\nspeed up the convergence rate of deterministic optimization, also provides us\nnew ways of designing non-uniform and aggressive moving average schemes in\nstochastic optimization. Finally, we present some heuristics that help to\nimplement adaptive accelerated stochastic gradient descent methods and to\nfurther improve their practical performance for machine learning and deep\nlearning.","url_abs":"http://arxiv.org/abs/1810.00553v1","url_pdf":"http://arxiv.org/pdf/1810.00553v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimal-adaptive-and-accelerated-stochastic","repo_url":"https://github.com/severilov/A2Grad_optimizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}