{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/imagenet-training-in-minutes","title":"ImageNet Training in Minutes","arxiv_id":"1709.05011","date":"2017-09-14","proceeding":null,"authors":["Yang You","Zhao Zhang","Cho-Jui Hsieh","James Demmel","Kurt Keutzer"],"abstract":"Finishing 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU\ntakes 14 days. This training requires 10^18 single precision operations in\ntotal. On the other hand, the world's current fastest supercomputer can finish\n2 * 10^17 single precision operations per second (Dongarra et al 2017,\nhttps://www.top500.org/lists/2017/06/). If we can make full use of the\nsupercomputer for DNN training, we should be able to finish the 90-epoch\nResNet-50 training in one minute. However, the current bottleneck for fast DNN\ntraining is in the algorithm level. Specifically, the current batch size (e.g.\n512) is too small to make efficient use of many processors. For large-scale DNN\ntraining, we focus on using large-batch data-parallelism synchronous SGD\nwithout losing accuracy in the fixed epochs. The LARS algorithm (You, Gitman,\nGinsburg, 2017, arXiv:1708.03888) enables us to scale the batch size to\nextremely large case (e.g. 32K). We finish the 100-epoch ImageNet training with\nAlexNet in 11 minutes on 1024 CPUs. About three times faster than Facebook's\nresult (Goyal et al 2017, arXiv:1706.02677), we finish the 90-epoch ImageNet\ntraining with ResNet-50 in 20 minutes on 2048 KNLs without losing accuracy.\nState-of-the-art ImageNet training speed with ResNet-50 is 74.9% top-1 test\naccuracy in 15 minutes. We got 74.9% top-1 test accuracy in 64 epochs, which\nonly needs 14 minutes. Furthermore, when we increase the batch size to above\n16K, our accuracy is much higher than Facebook's on corresponding batch sizes.\nOur source code is available upon request.","url_abs":"http://arxiv.org/abs/1709.05011v10","url_pdf":"http://arxiv.org/pdf/1709.05011v10.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"imagenet-training-in-minutes","repo_url":"https://github.com/fuentesdt/livermask","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"16k","task_name":"16k"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"2048","task_name":"Playing the Game of 2048"}],"methods":[{"method_slug":"lars","method_name":"LARS"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1709.05011","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}