{"url":"/method/skipinit","slug":"skipinit","name":"SkipInit","full_name":"SkipInit","full_name_withheld":false,"description_markdown":"**SkipInit** is a method that aims to allow [normalization](https://paperswithcode.com/methods/category/normalization)-free training of neural networks by downscaling [residual branches](https://paperswithcode.com/method/residual-block) at initialization.  This is achieved by including a learnable scalar multiplier at the end of each residual branch, initialized to $\\alpha$.\r\n\r\nThe method is motivated by theoretical findings that [batch normalization](https://paperswithcode.com/method/batch-normalization) downscales the hidden activations on the residual branch by a factor on the order of the square root of the network depth (at initialization). Therefore, as the depth of a residual network is increased, the residual blocks are increasingly dominated by the [skip connection](https://paperswithcode.com/method/residual-connection), which drives the functions computed by residual blocks closer to the identity, preserving signal propagation and ensuring well-behaved gradients. This leads to the proposed method which can achieve this property through an [initialization](https://paperswithcode.com/methods/category/initialization) strategy rather than a [normalization](https://paperswithcode.com/methods/category/normalization) strategy.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks","paper":"/paper/batch-normalization-biases-deep-residual","first_author":"Soham De","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/batch-normalization-biases-deep-residual"},"source":{"url":"https://arxiv.org/abs/2002.10444v3","title":"Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Initialization","url":"/methods/category/initialization","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":null,"title":"Dynamical Isometry for Residual Networks","date":"2022-10-05","arxiv_id":"2210.02411","n_code_links":0,"syntology":null},{"paper":"/paper/batch-normalization-biases-deep-residual","title":"Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks","date":"2020-02-24","arxiv_id":"2002.10444","n_code_links":0,"syntology":null}],"papers_shown":2,"tasks":[{"task":"/task/diversity","name":"Diversity","papers":1}],"tasks_shown":1,"n_tasks":1,"usage_by_year":[{"year":"2020","papers":1},{"year":"2022","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/skipinit"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}