{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-small-batch-training-for-deep","title":"Revisiting Small Batch Training for Deep Neural Networks","arxiv_id":"1804.07612","date":"2018-04-20","proceeding":null,"authors":["Dominic Masters","Carlo Luschi"],"abstract":"Modern deep neural network training is typically based on mini-batch\nstochastic gradient optimization. While the use of large mini-batches increases\nthe available computational parallelism, small batch training has been shown to\nprovide improved generalization performance and allows a significantly smaller\nmemory footprint, which might also be exploited to improve machine throughput.\n  In this paper, we review common assumptions on learning rate scaling and\ntraining duration, as a basis for an experimental comparison of test\nperformance for different mini-batch sizes. We adopt a learning rate that\ncorresponds to a constant average weight update per gradient calculation (i.e.,\nper unit cost of computation), and point out that this results in a variance of\nthe weight updates that increases linearly with the mini-batch size $m$.\n  The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet\ndatasets show that increasing the mini-batch size progressively reduces the\nrange of learning rates that provide stable convergence and acceptable test\nperformance. On the other hand, small mini-batch sizes provide more up-to-date\ngradient calculations, which yields more stable and reliable training. The best\nperformance has been consistently obtained for mini-batch sizes between $m = 2$\nand $m = 32$, which contrasts with recent work advocating the use of mini-batch\nsizes in the thousands.","url_abs":"http://arxiv.org/abs/1804.07612v1","url_pdf":"http://arxiv.org/pdf/1804.07612v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-small-batch-training-for-deep","repo_url":"https://github.com/Jasmine216/Fine-grained-image-classification-Dog-Breeds","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"revisiting-small-batch-training-for-deep","repo_url":"https://github.com/TureganoJose/NeuralNetworkFramework","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"revisiting-small-batch-training-for-deep","repo_url":"https://github.com/shaoren-sim/yotsubanet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.07612","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}