{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/communication-trade-offs-for-local-sgd-with","title":"Communication trade-offs for Local-SGD with large step size","arxiv_id":null,"date":"2019-12-01","proceeding":"NeurIPS 2019 12","authors":["Aymeric Dieuleveut","Kumar Kshitij Patel"],"abstract":"Synchronous mini-batch SGD is state-of-the-art for large-scale distributed machine learning. However, in practice, its convergence is bottlenecked by slow communication rounds between worker nodes. A natural solution to reduce communication is to use the \\emph{``local-SGD''}  model in which the workers train their model independently and synchronize every once in a while. This algorithm improves the computation-communication trade-off but its convergence is not understood very well. We propose a non-asymptotic error analysis, which enables comparison to \\emph{one-shot averaging} i.e., a single communication round among independent workers, and \\emph{mini-batch averaging} i.e., communicating at every step. We also provide adaptive lower bounds on the communication frequency for large step-sizes ($ t^{-\\alpha} $, $ \\alpha\\in (1/2 , 1 ) $) and show that \\emph{Local-SGD} reduces communication by a factor of $O\\Big(\\frac{\\sqrt{T}}{P^{3/2}}\\Big)$, with $T$ the total number of gradients and $P$ machines.","url_abs":"http://papers.nips.cc/paper/9512-communication-trade-offs-for-local-sgd-with-large-step-size","url_pdf":"http://papers.nips.cc/paper/9512-communication-trade-offs-for-local-sgd-with-large-step-size.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"communication-trade-offs-for-local-sgd-with","repo_url":"https://github.com/kishinmh/Local-SGD","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}