{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/normalization-effects-on-deep-neural-networks","title":"Normalization effects on deep neural networks","arxiv_id":"2209.01018","date":"2022-09-02","proceeding":null,"authors":["Jiahui Yu","Konstantinos Spiliopoulos"],"abstract":"We study the effect of normalization on the layers of deep neural networks of feed-forward type. A given layer $i$ with $N_{i}$ hidden units is allowed to be normalized by $1/N_{i}^{\\gamma_{i}}$ with $\\gamma_{i}\\in[1/2,1]$ and we study the effect of the choice of the $\\gamma_{i}$ on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set. We find that in terms of variance of the neural network's output and test accuracy the best choice is to choose the $\\gamma_{i}$'s to be equal to one, which is the mean-field scaling. We also find that this is particularly true for the outer layer, in that the neural network's behavior is more sensitive in the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network's output. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose the learning rate hyperparameters. Such a choice guarantees that the neural network behaves in a statistically robust way as the $N_i$ grow to infinity.","url_abs":"https://arxiv.org/abs/2209.01018v1","url_pdf":"https://arxiv.org/pdf/2209.01018v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"normalization-effects-on-deep-neural-networks","repo_url":"https://github.com/kspiliopoulos/NENN_Deep","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.01018","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.01018"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kspiliopoulos/NENN_Deep","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5c403003338fa67b","entry":"determine_device","repo":"kspiliopoulos/NENN_Deep","repo_kind":"official","path":"functions.py","file_url":"https://github.com/kspiliopoulos/NENN_Deep/blob/HEAD/functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c403003338fa67b"}},{"code_sha256_prefix":"a1e0ec5d447e0b8e","entry":"load_mnist_data","repo":"kspiliopoulos/NENN_Deep","repo_kind":"official","path":"functions.py","file_url":"https://github.com/kspiliopoulos/NENN_Deep/blob/HEAD/functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1e0ec5d447e0b8e"}},{"code_sha256_prefix":"d9b4cdcd24c77672","entry":"run_model","repo":"kspiliopoulos/NENN_Deep","repo_kind":"official","path":"functions.py","file_url":"https://github.com/kspiliopoulos/NENN_Deep/blob/HEAD/functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9b4cdcd24c77672"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}