Methods › General › Initialization › SkipInit
SkipInit
Introduced by Soham De et al. in Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
SkipInit is a method that aims to allow normalization-free training of neural networks by downscaling residual branches at initialization. This is achieved by including a learnable scalar multiplier at the end of each residual branch, initialized to α.
The method is motivated by theoretical findings that batch normalization downscales the hidden activations on the residual branch by a factor on the order of the square root of the network depth (at initialization). Therefore, as the depth of a residual network is increased, the residual blocks are increasingly dominated by the skip connection, which drives the functions computed by residual blocks closer to the identity, preserving signal propagation and ensuring well-behaved gradients. This leads to the proposed method which can achieve this property through an initialization strategy rather than a normalization strategy.
Papers archive 2025-07-28
2 shown of 2, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Dynamical Isometry for Residual Networks 5 Oct 2022 · 0 repositories · arXiv:2210.02411
-
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks 24 Feb 2020 · 0 repositories · arXiv:2002.10444
Tasks archive 2025-07-28
1 task the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Diversity | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections