{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/qsgd-communication-efficient-sgd-via-gradient","title":"QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding","arxiv_id":"1610.02132","date":"2016-10-07","proceeding":"NeurIPS 2017 12","authors":["Dan Alistarh","Demjan Grubic","Jerry Li","Ryota Tomioka","Milan Vojnovic"],"abstract":"Parallel implementations of stochastic gradient descent (SGD) have received\nsignificant research attention, thanks to excellent scalability properties of\nthis algorithm, and to its efficiency in the context of training deep neural\nnetworks. A fundamental barrier for parallelizing large-scale SGD is the fact\nthat the cost of communicating the gradient updates between nodes can be very\nlarge. Consequently, lossy compression heuristics have been proposed, by which\nnodes only communicate quantized gradients. Although effective in practice,\nthese heuristics do not always provably converge, and it is not clear whether\nthey are optimal.\n  In this paper, we propose Quantized SGD (QSGD), a family of compression\nschemes which allow the compression of gradient updates at each node, while\nguaranteeing convergence under standard assumptions. QSGD allows the user to\ntrade off compression and convergence time: it can communicate a sublinear\nnumber of bits per iteration in the model dimension, and can achieve\nasymptotically optimal communication cost. We complement our theoretical\nresults with empirical data, showing that QSGD can significantly reduce\ncommunication cost, while being competitive with standard uncompressed\ntechniques on a variety of real tasks.\n  In particular, experiments show that gradient quantization applied to\ntraining of deep neural networks for image classification and automated speech\nrecognition can lead to significant reductions in communication cost, and\nend-to-end training time. For instance, on 16 GPUs, we are able to train a\nResNet-152 network on ImageNet 1.8x faster to full accuracy. Of note, we show\nthat there exist generic parameter settings under which all known network\narchitectures preserve or slightly improve their full accuracy when using\nquantization.","url_abs":"http://arxiv.org/abs/1610.02132v4","url_pdf":"http://arxiv.org/pdf/1610.02132v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"qsgd-communication-efficient-sgd-via-gradient","repo_url":"https://github.com/scottjiao/Gradient-Compression-Methods","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"qsgd-communication-efficient-sgd-via-gradient","repo_url":"https://github.com/shayantalaei/quafl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1610.02132","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1610.02132"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shayantalaei/quafl","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/scottjiao/Gradient-Compression-Methods","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"73b74bb9bce598c9","entry":"one_bit","repo":"scottjiao/Gradient-Compression-Methods","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/scottjiao/Gradient-Compression-Methods/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73b74bb9bce598c9"}},{"code_sha256_prefix":"8c0c528956fb38b9","entry":"quantize","repo":"scottjiao/Gradient-Compression-Methods","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/scottjiao/Gradient-Compression-Methods/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c0c528956fb38b9"}},{"code_sha256_prefix":"1a1db7022bea18af","entry":"sparse_randomized","repo":"scottjiao/Gradient-Compression-Methods","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/scottjiao/Gradient-Compression-Methods/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a1db7022bea18af"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}