{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/train-longer-generalize-better-closing-the","title":"Train longer, generalize better: closing the generalization gap in large batch training of neural networks","arxiv_id":"1705.08741","date":"2017-05-24","proceeding":"NeurIPS 2017 12","authors":["Elad Hoffer","Itay Hubara","Daniel Soudry"],"abstract":"Background: Deep learning models are typically trained using stochastic\ngradient descent or one of its variants. These methods update the weights using\ntheir gradient, estimated from a small fraction of the training data. It has\nbeen observed that when using large batch sizes there is a persistent\ndegradation in generalization performance - known as the \"generalization gap\"\nphenomena. Identifying the origin of this gap and closing it had remained an\nopen problem.\n  Contributions: We examine the initial high learning rate training phase. We\nfind that the weight distance from its initialization grows logarithmically\nwith the number of weight updates. We therefore propose a \"random walk on\nrandom landscape\" statistical model which is known to exhibit similar\n\"ultra-slow\" diffusion behavior. Following this hypothesis we conducted\nexperiments to show empirically that the \"generalization gap\" stems from the\nrelatively small number of updates rather than the batch size, and can be\ncompletely eliminated by adapting the training regime used. We further\ninvestigate different techniques to train models in the large-batch regime and\npresent a novel algorithm named \"Ghost Batch Normalization\" which enables\nsignificant decrease in the generalization gap without increasing the number of\nupdates. To validate our findings we conduct several additional experiments on\nMNIST, CIFAR-10, CIFAR-100 and ImageNet. Finally, we reassess common practices\nand beliefs concerning training of deep models and suggest they may not be\noptimal to achieve good generalization.","url_abs":"http://arxiv.org/abs/1705.08741v2","url_pdf":"http://arxiv.org/pdf/1705.08741v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"train-longer-generalize-better-closing-the","repo_url":"https://github.com/eladhoffer/bigBatch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1705.08741","atlas_url":"https://app.syntology.ai/?focus=1705.08741","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}