{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-training-without-depth-limits-batch","title":"Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion","arxiv_id":"2310.02012","date":"2023-10-03","proceeding":null,"authors":["Alexandru Meterez","Amir Joudaki","Francesco Orabona","Alexander Immer","Gunnar Rätsch","Hadi Daneshmand"],"abstract":"Normalization layers are one of the key building blocks for deep neural networks. Several theoretical studies have shown that batch normalization improves the signal propagation, by avoiding the representations from becoming collinear across the layers. However, results on mean-field theory of batch normalization also conclude that this benefit comes at the expense of exploding gradients in depth. Motivated by these two aspects of batch normalization, in this study we pose the following question: \"Can a batch-normalized network keep the optimal signal propagation properties, but avoid exploding gradients?\" We answer this question in the affirmative by giving a particular construction of an Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded gradients at any depth. Based on Weingarten calculus, we develop a rigorous and non-asymptotic theory for this constructed MLP that gives a precise characterization of forward signal propagation, while proving that gradients remain bounded for linearly independent input samples, which holds in most practical settings. Inspired by our theory, we also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.","url_abs":"https://arxiv.org/abs/2310.02012v1","url_pdf":"https://arxiv.org/pdf/2310.02012v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-training-without-depth-limits-batch","repo_url":"https://github.com/alexandrumeterez/bngrad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.02012","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.02012"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alexandrumeterez/bngrad","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"58d2ba2db914e81d","entry":"cosine","repo":"alexandrumeterez/bngrad","repo_kind":"official","path":"src/modules/utils.py","file_url":"https://github.com/alexandrumeterez/bngrad/blob/HEAD/src/modules/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58d2ba2db914e81d"}},{"code_sha256_prefix":"0e1be474d47a0161","entry":"dataset_to_tensors","repo":"alexandrumeterez/bngrad","repo_kind":"official","path":"src/modules/data_utils.py","file_url":"https://github.com/alexandrumeterez/bngrad/blob/HEAD/src/modules/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e1be474d47a0161"}},{"code_sha256_prefix":"c3c6ce1d9150df1d","entry":"generate_matrix_close_to_isometry","repo":"alexandrumeterez/bngrad","repo_kind":"official","path":"src/modules/utils.py","file_url":"https://github.com/alexandrumeterez/bngrad/blob/HEAD/src/modules/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3c6ce1d9150df1d"}},{"code_sha256_prefix":"bfa386df8b0a370c","entry":"generate_matrix_far_from_isometry","repo":"alexandrumeterez/bngrad","repo_kind":"official","path":"src/modules/utils.py","file_url":"https://github.com/alexandrumeterez/bngrad/blob/HEAD/src/modules/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfa386df8b0a370c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}