{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omnivore-an-optimizer-for-multi-device-deep","title":"Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs","arxiv_id":"1606.04487","date":"2016-06-14","proceeding":null,"authors":["Stefan Hadjis","Ce Zhang","Ioannis Mitliagkas","Dan Iter","Christopher Ré"],"abstract":"We study the factors affecting training time in multi-device deep learning\nsystems. Given a specification of a convolutional neural network, our goal is\nto minimize the time to train this model on a cluster of commodity CPUs and\nGPUs. We first focus on the single-node setting and show that by using standard\nbatching and data-parallel techniques, throughput can be improved by at least\n5.5x over state-of-the-art systems on CPUs. This ensures an end-to-end training\nspeed directly proportional to the throughput of a device regardless of its\nunderlying hardware, allowing each node in the cluster to be treated as a black\nbox. Our second contribution is a theoretical and empirical study of the\ntradeoffs affecting end-to-end training time in a multiple-device setting. We\nidentify the degree of asynchronous parallelization as a key factor affecting\nboth hardware and statistical efficiency. We see that asynchrony can be viewed\nas introducing a momentum term. Our results imply that tuning momentum is\ncritical in asynchronous parallel configurations, and suggest that published\nresults that have not been fully tuned might report suboptimal performance for\nsome configurations. For our third contribution, we use our novel understanding\nof the interaction between system and optimization dynamics to provide an\nefficient hyperparameter optimizer. Our optimizer involves a predictive model\nfor the total time to convergence and selects an allocation of resources to\nminimize that time. We demonstrate that the most popular distributed deep\nlearning systems fall within our tradeoff space, but do not optimize within the\nspace. By doing this optimization, our prototype runs 1.9x to 12x faster than\nthe fastest state-of-the-art systems.","url_abs":"http://arxiv.org/abs/1606.04487v4","url_pdf":"http://arxiv.org/pdf/1606.04487v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omnivore-an-optimizer-for-multi-device-deep","repo_url":"https://github.com/HazyResearch/Omnivore","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1606.04487","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}