{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-understanding-regularization-in-batch","title":"Towards Understanding Regularization in Batch Normalization","arxiv_id":"1809.00846","date":"2018-09-04","proceeding":"ICLR 2019 5","authors":["Ping Luo","Xinjiang Wang","Wenqi Shao","Zhanglin Peng"],"abstract":"Batch Normalization (BN) improves both convergence and generalization in\ntraining neural networks. This work understands these phenomena theoretically.\nWe analyze BN by using a basic block of neural networks, consisting of a kernel\nlayer, a BN layer, and a nonlinear activation function. This basic network\nhelps us understand the impacts of BN in three aspects. First, by viewing BN as\nan implicit regularizer, BN can be decomposed into population normalization\n(PN) and gamma decay as an explicit regularization. Second, learning dynamics\nof BN and the regularization show that training converged with large maximum\nand effective learning rate. Third, generalization of BN is explored by using\nstatistical mechanics. Experiments demonstrate that BN in convolutional neural\nnetworks share the same traits of regularization as the above analyses.","url_abs":"http://arxiv.org/abs/1809.00846v4","url_pdf":"http://arxiv.org/pdf/1809.00846v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-understanding-regularization-in-batch","repo_url":"https://github.com/darshansiddu01/CNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1809.00846","atlas_url":"https://app.syntology.ai/?focus=1809.00846","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}