{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-to-initialize-your-network-robust","title":"How to Initialize your Network? Robust Initialization for WeightNorm & ResNets","arxiv_id":"1906.02341","date":"2019-06-05","proceeding":"NeurIPS 2019 12","authors":["Devansh Arpit","Victor Campos","Yoshua Bengio"],"abstract":"Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy. Similarly, initialization for ResNets have also been studied for un-normalized networks and often under simplified settings ignoring the shortcut connection. To address these issues, we propose a novel parameter initialization strategy that avoids explosion/vanishment of information across layers for weight normalized networks with and without residual connections. The proposed strategy is based on a theoretical analysis using mean field approximation. We run over 2,500 experiments and evaluate our proposal on image datasets showing that the proposed initialization outperforms existing initialization methods in terms of generalization performance, robustness to hyper-parameter values and variance between seeds, especially when networks get deeper in which case existing methods fail to even start training. Finally, we show that using our initialization in conjunction with learning rate warmup is able to reduce the gap between the performance of weight normalized and batch normalized networks.","url_abs":"https://arxiv.org/abs/1906.02341v2","url_pdf":"https://arxiv.org/pdf/1906.02341v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-to-initialize-your-network-robust","repo_url":"https://github.com/victorcampos7/weightnorm-init","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"how-to-initialize-your-network-robust","repo_url":"https://github.com/devansharpit/overparametrization_benefits","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"weight-normalization","method_name":"Weight Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1906.02341","atlas_url":"https://app.syntology.ai/?focus=1906.02341","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.02341"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/devansharpit/overparametrization_benefits","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/victorcampos7/weightnorm-init","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"557aa71e0848bed0","entry":"choose_left","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/schedules.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/schedules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"557aa71e0848bed0"}},{"code_sha256_prefix":"17e87ddd51cd5ba2","entry":"create_optimizer","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/graph_def.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/graph_def.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"17e87ddd51cd5ba2"}},{"code_sha256_prefix":"fc6b085641ffab5e","entry":"data_dependent_weight_norm_g_init","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/models/init.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/models/init.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"fc6b085641ffab5e"}},{"code_sha256_prefix":"1cec52b5952e109a","entry":"get_config_path","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/misc.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1cec52b5952e109a"}},{"code_sha256_prefix":"a23f08f7da4b2c36","entry":"get_input_size","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/graph_def.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/graph_def.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a23f08f7da4b2c36"}},{"code_sha256_prefix":"0f17d5f12485b857","entry":"get_log_path","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/misc.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"0f17d5f12485b857"}},{"code_sha256_prefix":"ac5e4caf2614e958","entry":"get_model_dir","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/misc.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ac5e4caf2614e958"}},{"code_sha256_prefix":"bdbb021a27680d79","entry":"hessian_spectral_norm_approx","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/hessian.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/hessian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"bdbb021a27680d79"}},{"code_sha256_prefix":"6b4b313f3974194b","entry":"init_and_maybe_normalize_layer","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/models/init.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/models/init.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"6b4b313f3974194b"}},{"code_sha256_prefix":"5908b0de70ae8316","entry":"linear_interpolation","repo":"victorcampos7/weightnorm-init","repo_kind":"official","path":"src/util/schedules.py","file_url":"https://github.com/victorcampos7/weightnorm-init/blob/HEAD/src/util/schedules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5908b0de70ae8316"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}