{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dont-decay-the-learning-rate-increase-the","title":"Don't Decay the Learning Rate, Increase the Batch Size","arxiv_id":"1711.00489","date":"2017-11-01","proceeding":"ICLR 2018 1","authors":["Samuel L. Smith","Pieter-Jan Kindermans","Chris Ying","Quoc V. Le"],"abstract":"It is common practice to decay the learning rate. Here we show one can\nusually obtain the same learning curve on both training and test sets by\ninstead increasing the batch size during training. This procedure is successful\nfor stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum,\nand Adam. It reaches equivalent test accuracies after the same number of\ntraining epochs, but with fewer parameter updates, leading to greater\nparallelism and shorter training times. We can further reduce the number of\nparameter updates by increasing the learning rate $\\epsilon$ and scaling the\nbatch size $B \\propto \\epsilon$. Finally, one can increase the momentum\ncoefficient $m$ and scale $B \\propto 1/(1-m)$, although this tends to slightly\nreduce the test accuracy. Crucially, our techniques allow us to repurpose\nexisting training schedules for large batch training with no hyper-parameter\ntuning. We train ResNet-50 on ImageNet to $76.1\\%$ validation accuracy in under\n30 minutes.","url_abs":"http://arxiv.org/abs/1711.00489v2","url_pdf":"http://arxiv.org/pdf/1711.00489v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dont-decay-the-learning-rate-increase-the","repo_url":"https://github.com/M-S-ALAM/Mercari-Price-Suggestion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"dont-decay-the-learning-rate-increase-the","repo_url":"https://github.com/rbkim1990/capstone-age-estimation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"dont-decay-the-learning-rate-increase-the","repo_url":"https://github.com/sukjunhwang/IFC","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1711.00489","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}