{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-distributed-synchronous-sgd","title":"Revisiting Distributed Synchronous SGD","arxiv_id":"1604.00981","date":"2016-04-04","proceeding":null,"authors":["Jianmin Chen","Xinghao Pan","Rajat Monga","Samy Bengio","Rafal Jozefowicz"],"abstract":"Distributed training of deep learning models on large-scale training data is\ntypically conducted with asynchronous stochastic optimization to maximize the\nrate of updates, at the cost of additional noise introduced from asynchrony. In\ncontrast, the synchronous approach is often thought to be impractical due to\nidle time wasted on waiting for straggling workers. We revisit these\nconventional beliefs in this paper, and examine the weaknesses of both\napproaches. We demonstrate that a third approach, synchronous optimization with\nbackup workers, can avoid asynchronous noise while mitigating for the worst\nstragglers. Our approach is empirically validated and shown to converge faster\nand to better test accuracies.","url_abs":"http://arxiv.org/abs/1604.00981v3","url_pdf":"http://arxiv.org/pdf/1604.00981v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-distributed-synchronous-sgd","repo_url":"https://github.com/Robinatp/Tensorflow_Model_Inception","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"revisiting-distributed-synchronous-sgd","repo_url":"https://github.com/epfml/byzantine-robust-optimizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"revisiting-distributed-synchronous-sgd","repo_url":"https://github.com/tensorflow/models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"revisiting-distributed-synchronous-sgd","repo_url":"https://github.com/tensorflow/models/tree/master/research/inception","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1604.00981","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}