{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hierarchical-cost-analysis-for-distributed-dl","title":"Hierarchical Cost Analysis for Distributed DL","arxiv_id":null,"date":"2021-05-17","proceeding":"IEEE International Parallel and Distributed Processing Symposium Workshops 2021 5","authors":["Haoran Wang"],"abstract":"Deep Learning (DL) developed rapidly during the past decade. DNN models become larger and more\r\ncomplex. Increasing size of datasets and models, requires efficient distributed approaches. Different\r\nparallelism strategies result in different performance depending on the structure of DNNs. In order to\r\nobtain better performance and overcome the memory restrictions, Hybrid Parallelisms (HP), which\r\napply different basic parallelism strategies on different parts of DNNs, are also explored. However,\r\ndifferent parallelisms bring about mixed extra costs which are difficult to distinguish and evaluate. It is\r\ncrucial to provide an approach which could clearly evaluate the costs caused by parallelisms and\r\nsystematically find efficient hybrid strategies. Current approaches only consider one or two kinds of\r\nparallelisms.\r\nIn this work, we firstly present the training process of DNNs and give the explanations of the DNN\r\nnotions. Three basic parallelism strategies (DP: data parallelism, OP: operator parallelism, PP: pipeline\r\nparallelism) are introduced and their relative merits are compared. The computation and communication\r\nare naturally distinguished when only considering the HP of DP/PP. DP determines the synchronous\r\ncommunication, but has no effect on forward/backward propagation (FPG/BPG), PP causes a bubble and\r\nlittle communication but will not affect the synchronization. Fundamental cost analysis can be easily\r\napplied. However, when taking OP into consideration, the FPG/BPG are changed to a mixed process of\r\ncommunication and computation where the previous cost analysis is no longer suitable. Existing\r\napproaches have not created a concrete execution model for distributed DNN training, but simply\r\nevaluate the total cost of computation and communication. Without distinguishing different kinds of\r\ncommunication, the proper HP can only be obtained through traversal or tuning, and loss further\r\noptimization opportunities.\r\nIn order to formalize the behaviors of the HP in distributed DL and quantitatively evaluate the cost caused\r\nby HP, we are studying Bridging DL composed by a double-level execution model associated with a\r\nsymbolic cost model. The double-level execution model is used to explore the details of the HPs. The\r\ntraining process of the whole DNN model is abstracted by a super-step while the training of an operator is\r\nabstracted by a micro-step. With the two steps, the mixed communication and computation caused by\r\nthree basic parallelism strategies are properly separated and the training process is clearly described.\r\nBased on the double-level execution model, the cost model of distributed DL helps to choose efficient HP\r\nstrategies. Details of the cost model can be found in the poster and extended abstracts.\r\nTo conclude, Bridging DL enables systematical HP searching for distributed DNN training and provides a\r\ngood opportunity for further analyzing and optimizations. Besides, Bridging DL also helps to guide the\r\nDNN framework for code generation of HP. A preliminary model (for micro-step) has been implemented\r\non MindSpore. The following parts will be implemented and evaluated.","url_abs":"https://www.computer.org/csdl/pds/api/csdl/proceedings/download-article/1uHgBhiabok/pdf","url_pdf":"https://www.computer.org/csdl/pds/api/csdl/proceedings/download-article/1uHgBhiabok/pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hierarchical-cost-analysis-for-distributed-dl","repo_url":"https://github.com/yangyucheng000/ConvLSTM_MindSpore","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}