{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-distributed-synchronous-sgd-algorithm-with","title":"A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks","arxiv_id":"1901.04359","date":"2019-01-14","proceeding":null,"authors":["Shaohuai Shi","Qiang Wang","Kaiyong Zhao","Zhenheng Tang","Yuxin Wang","Xiang Huang","Xiaowen Chu"],"abstract":"Distributed synchronous stochastic gradient descent (S-SGD) has been widely\nused in training large-scale deep neural networks (DNNs), but it typically\nrequires very high communication bandwidth between computational workers (e.g.,\nGPUs) to exchange gradients iteratively. Recently, Top-$k$ sparsification\ntechniques have been proposed to reduce the volume of data to be exchanged\namong workers. Top-$k$ sparsification can zero-out a significant portion of\ngradients without impacting the model convergence. However, the sparse\ngradients should be transferred with their irregular indices, which makes the\nsparse gradients aggregation difficult. Current methods that use AllGather to\naccumulate the sparse gradients have a communication complexity of $O(kP)$,\nwhere $P$ is the number of workers, which is inefficient on low bandwidth\nnetworks with a large number of workers. We observe that not all top-$k$\ngradients from $P$ workers are needed for the model update, and therefore we\npropose a novel global Top-$k$ (gTop-$k$) sparsification mechanism to address\nthe problem. Specifically, we choose global top-$k$ largest absolute values of\ngradients from $P$ workers, instead of accumulating all local top-$k$ gradients\nto update the model in each iteration. The gradient aggregation method based on\ngTop-$k$ sparsification reduces the communication complexity from $O(kP)$ to\n$O(k\\log P)$. Through extensive experiments on different DNNs, we verify that\ngTop-$k$ S-SGD has nearly consistent convergence performance with S-SGD, and it\nhas only slight degradations on generalization performance. In terms of scaling\nefficiency, we evaluate gTop-$k$ on a cluster with 32 GPU machines which are\ninterconnected with 1 Gbps Ethernet. The experimental results show that our\nmethod achieves $2.7-12\\times$ higher scaling efficiency than S-SGD and\n$1.1-1.7\\times$ improvement than the existing Top-$k$ S-SGD.","url_abs":"http://arxiv.org/abs/1901.04359v2","url_pdf":"http://arxiv.org/pdf/1901.04359v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-distributed-synchronous-sgd-algorithm-with","repo_url":"https://github.com/hclhkbu/gtopkssgd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}