{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/demystifying-parallel-and-distributed-deep","title":"Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis","arxiv_id":"1802.09941","date":"2018-02-26","proceeding":null,"authors":["Tal Ben-Nun","Torsten Hoefler"],"abstract":"Deep Neural Networks (DNNs) are becoming an important tool in modern\ncomputing applications. Accelerating their training is a major challenge and\ntechniques range from distributed algorithms to low-level circuit design. In\nthis survey, we describe the problem from a theoretical perspective, followed\nby approaches for its parallelization. We present trends in DNN architectures\nand the resulting implications on parallelization strategies. We then review\nand model the different types of concurrency in DNNs: from the single operator,\nthrough parallelism in network inference and training, to distributed deep\nlearning. We discuss asynchronous stochastic optimization, distributed system\narchitectures, communication schemes, and neural architecture search. Based on\nthose approaches, we extrapolate potential directions for parallelism in deep\nlearning.","url_abs":"http://arxiv.org/abs/1802.09941v2","url_pdf":"http://arxiv.org/pdf/1802.09941v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"demystifying-parallel-and-distributed-deep","repo_url":"https://github.com/ericyang789/Parallel-Compute-Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"},{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}