{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparsified-sgd-with-memory","title":"Sparsified SGD with Memory","arxiv_id":"1809.07599","date":"2018-09-20","proceeding":"NeurIPS 2018 12","authors":["Sebastian U. Stich","Jean-Baptiste Cordonnier","Martin Jaggi"],"abstract":"Huge scale machine learning problems are nowadays tackled by distributed\noptimization algorithms, i.e. algorithms that leverage the compute power of\nmany devices for training. The communication overhead is a key bottleneck that\nhinders perfect scalability. Various recent works proposed to use quantization\nor sparsification techniques to reduce the amount of data that needs to be\ncommunicated, for instance by only sending the most significant entries of the\nstochastic gradient (top-k sparsification). Whilst such schemes showed very\npromising performance in practice, they have eluded theoretical analysis so\nfar.\n  In this work we analyze Stochastic Gradient Descent (SGD) with\nk-sparsification or compression (for instance top-k or random-k) and show that\nthis scheme converges at the same rate as vanilla SGD when equipped with error\ncompensation (keeping track of accumulated errors in memory). That is,\ncommunication can be reduced by a factor of the dimension of the problem\n(sometimes even more) whilst still converging at the same rate. We present\nnumerical experiments to illustrate the theoretical findings and the better\nscalability for distributed applications.","url_abs":"http://arxiv.org/abs/1809.07599v2","url_pdf":"http://arxiv.org/pdf/1809.07599v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparsified-sgd-with-memory","repo_url":"https://github.com/epfml/sparsifiedSGD","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"distributed-optimization","task_name":"Distributed Optimization"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1809.07599","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.07599"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/epfml/sparsifiedSGD","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"96442fc743ff7faf","entry":"quantize","repo":"epfml/sparsifiedSGD","repo_kind":"official","path":"qsgd.py","file_url":"https://github.com/epfml/sparsifiedSGD/blob/HEAD/qsgd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96442fc743ff7faf"}},{"code_sha256_prefix":"438c8b99e9634673","entry":"to_percent","repo":"epfml/sparsifiedSGD","repo_kind":"official","path":"utils.py","file_url":"https://github.com/epfml/sparsifiedSGD/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"438c8b99e9634673"}},{"code_sha256_prefix":"70bc7a292e413fb6","entry":"unpickle_dir","repo":"epfml/sparsifiedSGD","repo_kind":"official","path":"utils.py","file_url":"https://github.com/epfml/sparsifiedSGD/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70bc7a292e413fb6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}