{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-decentralized-algorithms-outperform","title":"Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent","arxiv_id":"1705.09056","date":"2017-05-25","proceeding":"NeurIPS 2017 12","authors":["Xiangru Lian","Ce Zhang","huan zhang","Cho-Jui Hsieh","Wei zhang","Ji Liu"],"abstract":"Most distributed machine learning systems nowadays, including TensorFlow and\nCNTK, are built in a centralized fashion. One bottleneck of centralized\nalgorithms lies on high communication cost on the central node. Motivated by\nthis, we ask, can decentralized algorithms be faster than its centralized\ncounterpart?\n  Although decentralized PSGD (D-PSGD) algorithms have been studied by the\ncontrol community, existing analysis and theory do not show any advantage over\ncentralized PSGD (C-PSGD) algorithms, simply assuming the application scenario\nwhere only the decentralized network is available. In this paper, we study a\nD-PSGD algorithm and provide the first theoretical analysis that indicates a\nregime in which decentralized algorithms might outperform centralized\nalgorithms for distributed stochastic gradient descent. This is because D-PSGD\nhas comparable total computational complexities to C-PSGD but requires much\nless communication cost on the busiest node. We further conduct an empirical\nstudy to validate our theoretical analysis across multiple frameworks (CNTK and\nTorch), different network configurations, and computation platforms up to 112\nGPUs. On network configurations with low bandwidth or high latency, D-PSGD can\nbe up to one order of magnitude faster than its well-optimized centralized\ncounterparts.","url_abs":"http://arxiv.org/abs/1705.09056v5","url_pdf":"http://arxiv.org/pdf/1705.09056v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-decentralized-algorithms-outperform","repo_url":"https://github.com/VhalPurohit/290s","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"can-decentralized-algorithms-outperform","repo_url":"https://github.com/aparna-aketi/d_psgd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"can-decentralized-algorithms-outperform","repo_url":"https://github.com/facebookresearch/stochastic_gradient_push","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1705.09056","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1705.09056"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/stochastic_gradient_push","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/VhalPurohit/290s","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aparna-aketi/d_psgd","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"aee4a2097410762b","entry":"train","repo":"aparna-aketi/d_psgd","repo_kind":"listed","path":"trainer.py","file_url":"https://github.com/aparna-aketi/d_psgd/blob/HEAD/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aee4a2097410762b"}},{"code_sha256_prefix":"5923c197179c7a44","entry":"validate","repo":"aparna-aketi/d_psgd","repo_kind":"listed","path":"trainer.py","file_url":"https://github.com/aparna-aketi/d_psgd/blob/HEAD/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5923c197179c7a44"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}