{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-accuracy-and-runtime-tradeoff-in","title":"Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study","arxiv_id":"1509.04210","date":"2015-09-14","proceeding":null,"authors":["Suyog Gupta","Wei zhang","Fei Wang"],"abstract":"This paper presents Rudra, a parameter server based distributed computing\nframework tuned for training large-scale deep neural networks. Using variants\nof the asynchronous stochastic gradient descent algorithm we study the impact\nof synchronization protocol, stale gradient updates, minibatch size, learning\nrates, and number of learners on runtime performance and model accuracy. We\nintroduce a new learning rate modulation strategy to counter the effect of\nstale gradients and propose a new synchronization protocol that can effectively\nbound the staleness in gradients, improve runtime performance and achieve good\nmodel accuracy. Our empirical investigation reveals a principled approach for\ndistributed training of neural networks: the mini-batch size per learner should\nbe reduced as more learners are added to the system to preserve the model\naccuracy. We validate this approach using commonly-used image classification\nbenchmarks: CIFAR10 and ImageNet.","url_abs":"http://arxiv.org/abs/1509.04210v3","url_pdf":"http://arxiv.org/pdf/1509.04210v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-accuracy-and-runtime-tradeoff-in","repo_url":"https://github.com/Farhad-n/MultiGPU_Study","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"distributed-computing","task_name":"Distributed Computing"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}