{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/experiments-on-parallel-training-of-deep","title":"Experiments on Parallel Training of Deep Neural Network using Model Averaging","arxiv_id":"1507.01239","date":"2015-07-05","proceeding":null,"authors":["Hang Su","Haoyu Chen"],"abstract":"In this work we apply model averaging to parallel training of deep neural\nnetwork (DNN). Parallelization is done in a model averaging manner. Data is\npartitioned and distributed to different nodes for local model updates, and\nmodel averaging across nodes is done every few minibatches. We use multiple\nGPUs for data parallelization, and Message Passing Interface (MPI) for\ncommunication between nodes, which allows us to perform model averaging\nfrequently without losing much time on communication. We investigate the\neffectiveness of Natural Gradient Stochastic Gradient Descent (NG-SGD) and\nRestricted Boltzmann Machine (RBM) pretraining for parallel training in\nmodel-averaging framework, and explore the best setups in term of different\nlearning rate schedules, averaging frequencies and minibatch sizes. It is shown\nthat NG-SGD and RBM pretraining benefits parameter-averaging based model\ntraining. On the 300h Switchboard dataset, a 9.3 times speedup is achieved\nusing 16 GPUs and 17 times speedup using 32 GPUs with limited decoding accuracy\nloss.","url_abs":"http://arxiv.org/abs/1507.01239v3","url_pdf":"http://arxiv.org/pdf/1507.01239v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"experiments-on-parallel-training-of-deep","repo_url":"https://github.com/Farhad-n/MultiGPU_Study","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1507.01239","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}