{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoupled-parallel-backpropagation-with","title":"Decoupled Parallel Backpropagation with Convergence Guarantee","arxiv_id":"1804.10574","date":"2018-04-27","proceeding":"ICML 2018 7","authors":["Zhouyuan Huo","Bin Gu","Qian Yang","Heng Huang"],"abstract":"Backpropagation algorithm is indispensable for the training of feedforward\nneural networks. It requires propagating error gradients sequentially from the\noutput layer all the way back to the input layer. The backward locking in\nbackpropagation algorithm constrains us from updating network layers in\nparallel and fully leveraging the computing resources. Recently, several\nalgorithms have been proposed for breaking the backward locking. However, their\nperformances degrade seriously when networks are deep. In this paper, we\npropose decoupled parallel backpropagation algorithm for deep learning\noptimization with convergence guarantee. Firstly, we decouple the\nbackpropagation algorithm using delayed gradients, and show that the backward\nlocking is removed when we split the networks into multiple modules. Then, we\nutilize decoupled parallel backpropagation in two stochastic methods and prove\nthat our method guarantees convergence to critical points for the non-convex\nproblem. Finally, we perform experiments for training deep convolutional neural\nnetworks on benchmark datasets. The experimental results not only confirm our\ntheoretical analysis, but also demonstrate that the proposed method can achieve\nsignificant speedup without loss of accuracy.","url_abs":"http://arxiv.org/abs/1804.10574v3","url_pdf":"http://arxiv.org/pdf/1804.10574v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decoupled-parallel-backpropagation-with","repo_url":"https://github.com/slowbull/DDG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"decoupled-parallel-backpropagation-with","repo_url":"https://github.com/hwlee11/PipelinedASGD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"decoupled-parallel-backpropagation-with","repo_url":"https://github.com/unconst/MACH","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.10574","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.10574"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/unconst/MACH","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/slowbull/DDG","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hwlee11/PipelinedASGD","reach":{"status":"ok"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"fc57fc79a5bb73a2","entry":"resnet_ddg","repo":"slowbull/DDG","repo_kind":"official","path":"models/resnet_ddg.py","file_url":"https://github.com/slowbull/DDG/blob/HEAD/models/resnet_ddg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc57fc79a5bb73a2"}},{"code_sha256_prefix":"ecc8f88a101a61e5","entry":"resnet_ddg_110","repo":"slowbull/DDG","repo_kind":"official","path":"models/resnet_ddg.py","file_url":"https://github.com/slowbull/DDG/blob/HEAD/models/resnet_ddg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecc8f88a101a61e5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}