{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-learning-with-elastic-averaging-sgd","title":"Deep learning with Elastic Averaging SGD","arxiv_id":"1412.6651","date":"2014-12-20","proceeding":"NeurIPS 2015 12","authors":["Sixin Zhang","Anna Choromanska","Yann Lecun"],"abstract":"We study the problem of stochastic optimization for deep learning in the\nparallel computing environment under communication constraints. A new algorithm\nis proposed in this setting where the communication and coordination of work\namong concurrent processes (local workers), is based on an elastic force which\nlinks the parameters they compute with a center variable stored by the\nparameter server (master). The algorithm enables the local workers to perform\nmore exploration, i.e. the algorithm allows the local variables to fluctuate\nfurther from the center variable by reducing the amount of communication\nbetween local workers and the master. We empirically demonstrate that in the\ndeep learning setting, due to the existence of many local optima, allowing more\nexploration can lead to the improved performance. We propose synchronous and\nasynchronous variants of the new algorithm. We provide the stability analysis\nof the asynchronous variant in the round-robin scheme and compare it with the\nmore common parallelized method ADMM. We show that the stability of EASGD is\nguaranteed when a simple stability condition is satisfied, which is not the\ncase for ADMM. We additionally propose the momentum-based version of our\nalgorithm that can be applied in both synchronous and asynchronous settings.\nAsynchronous variant of the algorithm is applied to train convolutional neural\nnetworks for image classification on the CIFAR and ImageNet datasets.\nExperiments demonstrate that the new algorithm accelerates the training of deep\narchitectures compared to DOWNPOUR and other common baseline approaches and\nfurthermore is very communication efficient.","url_abs":"http://arxiv.org/abs/1412.6651v8","url_pdf":"http://arxiv.org/pdf/1412.6651v8.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/sixin-zh/mpiT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"torch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/JoeriHermans/dist-keras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/cerndb/dist-keras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/duanders/mpi_learn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/mila-udem/platoon","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/uoguelph-mlrg/Theano-MPI","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/valentinchelle/kerasOnSpark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/vlimant/NNLO","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"deep-learning-with-elastic-averaging-sgd","repo_url":"https://github.com/vlimant/mpi_learn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"admm","method_name":"ADMM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1412.6651","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1412.6651"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vlimant/mpi_learn","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mila-udem/platoon","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JoeriHermans/dist-keras","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/valentinchelle/kerasOnSpark","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sixin-zh/mpiT","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cerndb/dist-keras","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vlimant/NNLO","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/duanders/mpi_learn","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uoguelph-mlrg/Theano-MPI","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":12},"by_repo_kind":{"listed":{"samples":12,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1c0eb2292a1f7394","entry":"AverageSGD","repo":"mila-udem/platoon","repo_kind":"listed","path":"platoon/training/global_dynamics.py","file_url":"https://github.com/mila-udem/platoon/blob/HEAD/platoon/training/global_dynamics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c0eb2292a1f7394"}},{"code_sha256_prefix":"3f0bece70f1f844a","entry":"SumSGD","repo":"mila-udem/platoon","repo_kind":"listed","path":"platoon/training/global_dynamics.py","file_url":"https://github.com/mila-udem/platoon/blob/HEAD/platoon/training/global_dynamics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f0bece70f1f844a"}},{"code_sha256_prefix":"f8a4a97b4d0d03f9","entry":"assign_global_to_local","repo":"tmulc18/Distributed-TensorFlow-Guide","repo_kind":"listed","path":"AGN/AGN.py","file_url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide/blob/HEAD/AGN/AGN.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8a4a97b4d0d03f9"}},{"code_sha256_prefix":"cfe0509e6a9c0096","entry":"assign_global_to_local","repo":"tmulc18/Distributed-TensorFlow-Guide","repo_kind":"listed","path":"DOWNPOUR-Easy/DOWNPOUR.py","file_url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide/blob/HEAD/DOWNPOUR-Easy/DOWNPOUR.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfe0509e6a9c0096"}},{"code_sha256_prefix":"92949b36477b9bf0","entry":"assign_local_to_global","repo":"tmulc18/Distributed-TensorFlow-Guide","repo_kind":"listed","path":"AGN/AGN.py","file_url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide/blob/HEAD/AGN/AGN.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92949b36477b9bf0"}},{"code_sha256_prefix":"acd320b0a5508644","entry":"fetch_devices_for_host","repo":"mila-udem/platoon","repo_kind":"listed","path":"platoon/configparser.py","file_url":"https://github.com/mila-udem/platoon/blob/HEAD/platoon/configparser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acd320b0a5508644"}},{"code_sha256_prefix":"b62207d24c9391fe","entry":"get_global_variable_by_name","repo":"tmulc18/Distributed-TensorFlow-Guide","repo_kind":"listed","path":"AGN/AGN.py","file_url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide/blob/HEAD/AGN/AGN.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b62207d24c9391fe"}},{"code_sha256_prefix":"af0c2ca334ca8e29","entry":"get_variable_by_name","repo":"tmulc18/Distributed-TensorFlow-Guide","repo_kind":"listed","path":"DOWNPOUR-Easy/DOWNPOUR.py","file_url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide/blob/HEAD/DOWNPOUR-Easy/DOWNPOUR.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af0c2ca334ca8e29"}},{"code_sha256_prefix":"a8b21a10b799b5c9","entry":"get_variable_by_name","repo":"tmulc18/Distributed-TensorFlow-Guide","repo_kind":"listed","path":"DOWNPOUR/DOWNPOUR.py","file_url":"https://github.com/tmulc18/Distributed-TensorFlow-Guide/blob/HEAD/DOWNPOUR/DOWNPOUR.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8b21a10b799b5c9"}},{"code_sha256_prefix":"6985423314deb16d","entry":"launch_process","repo":"mila-udem/platoon","repo_kind":"listed","path":"platoon/util.py","file_url":"https://github.com/mila-udem/platoon/blob/HEAD/platoon/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6985423314deb16d"}},{"code_sha256_prefix":"1c10065ed878004e","entry":"mmap","repo":"mila-udem/platoon","repo_kind":"listed","path":"platoon/util.py","file_url":"https://github.com/mila-udem/platoon/blob/HEAD/platoon/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c10065ed878004e"}},{"code_sha256_prefix":"ee9496eca0245020","entry":"op_to_mpi","repo":"mila-udem/platoon","repo_kind":"listed","path":"platoon/util.py","file_url":"https://github.com/mila-udem/platoon/blob/HEAD/platoon/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee9496eca0245020"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}