{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/noise-adaptive-accelerated-stochastic-heavy","title":"(Accelerated) Noise-adaptive Stochastic Heavy-Ball Momentum","arxiv_id":"2401.06738","date":"2024-01-12","proceeding":null,"authors":["Anh Dang","Reza Babanezhad","Sharan Vaswani"],"abstract":"Stochastic heavy ball momentum (SHB) is commonly used to train machine learning models, and often provides empirical improvements over stochastic gradient descent. By primarily focusing on strongly-convex quadratics, we aim to better understand the theoretical advantage of SHB and subsequently improve the method. For strongly-convex quadratics, Kidambi et al. (2018) show that SHB (with a mini-batch of size $1$) cannot attain accelerated convergence, and hence has no theoretical benefit over SGD. They conjecture that the practical gain of SHB is a by-product of using larger mini-batches. We first substantiate this claim by showing that SHB can attain an accelerated rate when the mini-batch size is larger than a threshold $b^*$ that depends on the condition number $\\kappa$. Specifically, we prove that with the same step-size and momentum parameters as in the deterministic setting, SHB with a sufficiently large mini-batch size results in an $O\\left(\\exp(-\\frac{T}{\\sqrt{\\kappa}}) + \\sigma \\right)$ convergence when measuring the distance to the optimal solution in the $\\ell_2$ norm, where $T$ is the number of iterations and $\\sigma^2$ is the variance in the stochastic gradients. We prove a lower-bound which demonstrates that a $\\kappa$ dependence in $b^*$ is necessary. To ensure convergence to the minimizer, we design a noise-adaptive multi-stage algorithm that results in an $O\\left(\\exp\\left(-\\frac{T}{\\sqrt{\\kappa}}\\right) + \\frac{\\sigma}{\\sqrt{T}}\\right)$ rate when measuring the distance to the optimal solution in the $\\ell_2$ norm. We also consider the general smooth, strongly-convex setting and propose the first noise-adaptive SHB variant that converges to the minimizer at an $O(\\exp(-\\frac{T}{\\kappa}) + \\frac{\\sigma^2}{T})$ rate when measuring the distance to the optimal solution in the squared $\\ell_2$ norm. We empirically demonstrate the effectiveness of the proposed algorithms.","url_abs":"https://arxiv.org/abs/2401.06738v3","url_pdf":"https://arxiv.org/pdf/2401.06738v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"noise-adaptive-accelerated-stochastic-heavy","repo_url":"https://github.com/anh-dang/accelerated_noise_adaptive_shb","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}