{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/signsgd-with-majority-vote-is-communication","title":"signSGD with Majority Vote is Communication Efficient And Fault Tolerant","arxiv_id":"1810.05291","date":"2018-10-11","proceeding":"ICLR 2019 5","authors":["Jeremy Bernstein","Jia-Wei Zhao","Kamyar Azizzadenesheli","Anima Anandkumar"],"abstract":"Training neural networks on large datasets can be accelerated by distributing\nthe workload over a network of machines. As datasets grow ever larger, networks\nof hundreds or thousands of machines become economically viable. The time cost\nof communicating gradients limits the effectiveness of using such large machine\ncounts, as may the increased chance of network faults. We explore a\nparticularly simple algorithm for robust, communication-efficient\nlearning---signSGD. Workers transmit only the sign of their gradient vector to\na server, and the overall update is decided by a majority vote. This algorithm\nuses $32\\times$ less communication per iteration than full-precision,\ndistributed SGD. Under natural conditions verified by experiment, we prove that\nsignSGD converges in the large and mini-batch settings, establishing\nconvergence for a parameter regime of Adam as a byproduct. Aggregating sign\ngradients by majority vote means that no individual worker has too much power.\nWe prove that unlike SGD, majority vote is robust when up to 50% of workers\nbehave adversarially. The class of adversaries we consider includes as special\ncases those that invert or randomise their gradient estimate. On the practical\nside, we built our distributed training system in Pytorch. Benchmarking against\nthe state of the art collective communications library (NCCL), our\nframework---with the parameter server housed entirely on one machine---led to a\n25% reduction in time for training resnet50 on Imagenet when using 15 AWS\np3.2xlarge machines.","url_abs":"http://arxiv.org/abs/1810.05291v3","url_pdf":"http://arxiv.org/pdf/1810.05291v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"signsgd-with-majority-vote-is-communication","repo_url":"https://github.com/jasonakoun/signsgd-fault-tolerance","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"signsgd-with-majority-vote-is-communication","repo_url":"https://github.com/jxbz/signSGD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"signsgd-with-majority-vote-is-communication","repo_url":"https://github.com/MindSpore-scientific/code-13/tree/main/signSGD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"signsgd-with-majority-vote-is-communication","repo_url":"https://github.com/PermiJW/signSGD-with-Majority-Vote","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1810.05291","atlas_url":"https://app.syntology.ai/?focus=1810.05291","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1810.05291"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jxbz/signSGD","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PermiJW/signSGD-with-Majority-Vote","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-13/tree/main/signSGD","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jasonakoun/signsgd-fault-tolerance","reach":null}],"summary":{"ran":1},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"906e7d9fe3cf6fbc","entry":"eval_dist","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"906e7d9fe3cf6fbc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}