{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-batchnorm-towards-a-general","title":"Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning","arxiv_id":"2106.05956","date":"2021-06-10","proceeding":"NeurIPS 2021 12","authors":["Ekdeep Singh Lubana","Robert P. Dick","Hidenori Tanaka"],"abstract":"Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to explain its success. However, given the pursuit of alternative normalization layers, these properties need to be generalized so that any given layer's success/failure can be accurately predicted. In this work, we take a first step towards this goal by extending known properties of BatchNorm in randomly initialized deep neural networks (DNNs) to several recently proposed normalization layers. Our primary findings follow: (i) similar to BatchNorm, activations-based normalization layers can prevent exponential growth of activations in ResNets, but parametric techniques require explicit remedies; (ii) use of GroupNorm can ensure an informative forward propagation, with different samples being assigned dissimilar activations, but increasing group size results in increasingly indistinguishable activations for different samples, explaining slow convergence speed in models with LayerNorm; and (iii) small group sizes result in large gradient norm in earlier layers, hence explaining training instability issues in Instance Normalization and illustrating a speed-stability tradeoff in GroupNorm. Overall, our analysis reveals a unified set of mechanisms that underpin the success of normalization methods in deep learning, providing us with a compass to systematically explore the vast design space of DNN normalization layers.","url_abs":"https://arxiv.org/abs/2106.05956v4","url_pdf":"https://arxiv.org/pdf/2106.05956v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-batchnorm-towards-a-general","repo_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[{"method_slug":"instance-normalization","method_name":"Instance Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.05956","atlas_url":"https://app.syntology.ai/?focus=2106.05956","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.05956"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EkdeepSLubana/BeyondBatchNorm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d7ec010c29e813b","entry":"format_time","repo":"EkdeepSLubana/BeyondBatchNorm","repo_kind":"official","path":"config.py","file_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d7ec010c29e813b"}},{"code_sha256_prefix":"c49f6938185f2abd","entry":"get_convlayer","repo":"EkdeepSLubana/BeyondBatchNorm","repo_kind":"official","path":"models.py","file_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c49f6938185f2abd"}},{"code_sha256_prefix":"87820eaab88d91a4","entry":"get_model","repo":"EkdeepSLubana/BeyondBatchNorm","repo_kind":"official","path":"models.py","file_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87820eaab88d91a4"}},{"code_sha256_prefix":"ec890b40515047cc","entry":"get_norm_and_activ_layer","repo":"EkdeepSLubana/BeyondBatchNorm","repo_kind":"official","path":"models.py","file_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec890b40515047cc"}},{"code_sha256_prefix":"c1d6b60dc57679ee","entry":"group_mean","repo":"EkdeepSLubana/BeyondBatchNorm","repo_kind":"official","path":"layer_defs.py","file_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm/blob/HEAD/layer_defs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1d6b60dc57679ee"}},{"code_sha256_prefix":"f2f879b4407d0abc","entry":"group_std","repo":"EkdeepSLubana/BeyondBatchNorm","repo_kind":"official","path":"layer_defs.py","file_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm/blob/HEAD/layer_defs.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2f879b4407d0abc"}},{"code_sha256_prefix":"67edf5596d211c06","entry":"instance_std","repo":"EkdeepSLubana/BeyondBatchNorm","repo_kind":"official","path":"layer_defs.py","file_url":"https://github.com/EkdeepSLubana/BeyondBatchNorm/blob/HEAD/layer_defs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67edf5596d211c06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}