{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparse-communication-for-distributed-gradient","title":"Sparse Communication for Distributed Gradient Descent","arxiv_id":"1704.05021","date":"2017-04-17","proceeding":"EMNLP 2017 9","authors":["Alham Fikri Aji","Kenneth Heafield"],"abstract":"We make distributed stochastic gradient descent faster by exchanging sparse\nupdates instead of dense updates. Gradient updates are positively skewed as\nmost updates are near zero, so we map the 99% smallest updates (by absolute\nvalue) to zero then exchange sparse matrices. This method can be combined with\nquantization to further improve the compression. We explore different\nconfigurations and apply them to neural machine translation and MNIST image\nclassification tasks. Most configurations work on MNIST, whereas different\nconfigurations reduce convergence rate on the more complex translation task.\nOur experiments show that we can achieve up to 49% speed up on MNIST and 22% on\nNMT without damaging the final accuracy or BLEU.","url_abs":"http://arxiv.org/abs/1704.05021v2","url_pdf":"http://arxiv.org/pdf/1704.05021v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparse-communication-for-distributed-gradient","repo_url":"https://github.com/KaiyuanZh/censor","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1704.05021","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}