{"url":"/method/kaiming-initialization","slug":"kaiming-initialization","name":"Kaiming Initialization","full_name":"Kaiming Initialization","full_name_withheld":false,"description_markdown":"**Kaiming Initialization**, or **He Initialization**, is an initialization method for neural networks that takes into account the non-linearity of activation functions, such as [ReLU](https://paperswithcode.com/method/relu) activations.\r\n\r\nA proper initialization method should avoid reducing or magnifying the magnitudes of input signals exponentially. Using a derivation they work out that the condition to stop this happening is:\r\n\r\n$$\\frac{1}{2}n\\_{l}\\text{Var}\\left[w\\_{l}\\right] = 1 $$\r\n\r\nThis implies an initialization scheme of:\r\n\r\n$$ w\\_{l} \\sim \\mathcal{N}\\left(0,  2/n\\_{l}\\right)$$\r\n\r\nThat is, a zero-centered Gaussian with standard deviation of $\\sqrt{2/{n}\\_{l}}$ (variance shown in equation above). Biases are initialized at $0$.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"http://arxiv.org/abs/1502.01852v1","title":"Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/0adb5843766092fba584791af76383125fd0d01c/torch/nn/init.py#L389","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Initialization","url":"/methods/category/initialization","pwc_aliases":[]}],"n_papers_tagged":2931,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/circumventing-backdoor-space-via-weight","title":"Circumventing Backdoor Space via Weight Symmetry","date":"2025-06-09","arxiv_id":"2506.07467","n_code_links":1,"syntology":null},{"paper":null,"title":"Synthetic Speech Source Tracing using Metric Learning","date":"2025-06-03","arxiv_id":"2506.02590","n_code_links":0,"syntology":null},{"paper":null,"title":"OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning","date":"2025-05-31","arxiv_id":"2506.00338","n_code_links":0,"syntology":null},{"paper":null,"title":"PointODE: Lightweight Point Cloud Learning with Neural Ordinary Differential Equations on Edge","date":"2025-05-31","arxiv_id":"2506.00438","n_code_links":0,"syntology":null},{"paper":null,"title":"ACM-UNet: Adaptive Integration of CNNs and Mamba for Efficient Medical Image Segmentation","date":"2025-05-30","arxiv_id":"2505.24481","n_code_links":0,"syntology":null},{"paper":"/paper/optimal-weighted-convolution-for","title":"Optimal Weighted Convolution for Classification and Denosing","date":"2025-05-30","arxiv_id":"2505.24558","n_code_links":2,"syntology":null},{"paper":null,"title":"Stepsize anything: A unified learning rate schedule for budgeted-iteration training","date":"2025-05-30","arxiv_id":"2505.24452","n_code_links":0,"syntology":null},{"paper":null,"title":"Leveraging Diffusion Models for Synthetic Data Augmentation in Protein Subcellular Localization Classification","date":"2025-05-28","arxiv_id":"2505.22926","n_code_links":0,"syntology":null},{"paper":null,"title":"Intelligent Incident Hypertension Prediction in Obstructive Sleep Apnea","date":"2025-05-27","arxiv_id":"2505.20615","n_code_links":0,"syntology":null},{"paper":null,"title":"Lung Nodule Segmentation: Exploring Data Efficiency and Advanced Architectures","date":"2025-05-26","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Hierarchical-embedding autoencoder with a predictor (HEAP) as efficient architecture for learning long-term evolution of complex multi-scale physical systems","date":"2025-05-24","arxiv_id":"2505.18857","n_code_links":0,"syntology":null},{"paper":null,"title":"SW-ViT: A Spatio-Temporal Vision Transformer Network with Post Denoiser for Sequential Multi-Push Ultrasound Shear Wave Elastography","date":"2025-05-24","arxiv_id":"2505.18865","n_code_links":0,"syntology":null},{"paper":"/paper/dect-based-space-squeeze-method-for-multi","title":"DECT-based Space-Squeeze Method for Multi-Class Classification of Metastatic Lymph Nodes in Breast Cancer","date":"2025-05-23","arxiv_id":"2505.17528","n_code_links":1,"syntology":null},{"paper":null,"title":"EVM-Fusion: An Explainable Vision Mamba Architecture with Neural Algorithmic Fusion","date":"2025-05-23","arxiv_id":"2505.17367","n_code_links":0,"syntology":null},{"paper":null,"title":"Detailed Evaluation of Modern Machine Learning Approaches for Optic Plastics Sorting","date":"2025-05-22","arxiv_id":"2505.16513","n_code_links":0,"syntology":null},{"paper":null,"title":"SuperPure: Efficient Purification of Localized and Distributed Adversarial Patches via Super-Resolution GAN Models","date":"2025-05-22","arxiv_id":"2505.16318","n_code_links":0,"syntology":null},{"paper":null,"title":"Analysis of ABC Frontend Audio Systems for the NIST-SRE24","date":"2025-05-21","arxiv_id":"2505.15320","n_code_links":0,"syntology":null},{"paper":null,"title":"Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers","date":"2025-05-21","arxiv_id":"2505.15239","n_code_links":0,"syntology":null},{"paper":"/paper/ssps-self-supervised-positive-sampling-for","title":"SSPS: Self-Supervised Positive Sampling for Robust Self-Supervised Speaker Verification","date":"2025-05-20","arxiv_id":"2505.14561","n_code_links":1,"syntology":null},{"paper":"/paper/vulnerability-of-transfer-learned-neural","title":"Vulnerability of Transfer-Learned Neural Networks to Data Reconstruction Attacks in Small-Data Regime","date":"2025-05-20","arxiv_id":"2505.14323","n_code_links":1,"syntology":null},{"paper":"/paper/2505-11129","title":"PhiNet v2: A Mask-Free Brain-Inspired Vision Foundation Model from Video","date":"2025-05-16","arxiv_id":"2505.11129","n_code_links":1,"syntology":null},{"paper":null,"title":"CheX-DS: Improving Chest X-ray Image Classification with Ensemble Learning Based on DenseNet and Swin Transformer","date":"2025-05-16","arxiv_id":"2505.11168","n_code_links":0,"syntology":null},{"paper":"/paper/a-training-framework-for-optimal-and-stable","title":"A Training Framework for Optimal and Stable Training of Polynomial Neural Networks","date":"2025-05-16","arxiv_id":"2505.11589","n_code_links":1,"syntology":null},{"paper":null,"title":"The Influence of the Memory Capacity of Neural DDEs on the Universal Approximation Property","date":"2025-05-12","arxiv_id":"2505.07244","n_code_links":0,"syntology":null},{"paper":null,"title":"Towards Understanding Deep Learning Model in Image Recognition via Coverage Test","date":"2025-05-12","arxiv_id":"2505.08814","n_code_links":0,"syntology":null},{"paper":"/paper/reproducing-and-improving-chexnet-deep","title":"Reproducing and Improving CheXNet: Deep Learning for Chest X-ray Disease Classification","date":"2025-05-10","arxiv_id":"2505.06646","n_code_links":1,"syntology":null},{"paper":null,"title":"Achieving 3D Attention via Triplet Squeeze and Excitation Block","date":"2025-05-09","arxiv_id":"2505.05943","n_code_links":0,"syntology":null},{"paper":null,"title":"A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition","date":"2025-05-08","arxiv_id":"2505.05148","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep residual learning with product units","date":"2025-05-07","arxiv_id":"2505.04397","n_code_links":0,"syntology":null},{"paper":null,"title":"DOTA: Deformable Optimized Transformer Architecture for End-to-End Text Recognition with Retrieval-Augmented Generation","date":"2025-05-07","arxiv_id":"2505.04175","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":453},{"task":"/task/image-classification","name":"image-classification","papers":349},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":285},{"task":"/task/classification","name":"General Classification","papers":229},{"task":"/task/object-detection","name":"Object Detection","papers":211},{"task":"/task/classification-1","name":"Classification","papers":203},{"task":"/task/segmentation","name":"Segmentation","papers":193},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":184},{"task":"/task/object-detection-1","name":"object-detection","papers":177},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":159},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":148},{"task":"/task/representation-learning","name":"Representation Learning","papers":144},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":128},{"task":"/task/deep-learning","name":"Deep Learning","papers":115},{"task":"/task/object","name":"Object","papers":80},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":76},{"task":null,"name":"GPU","papers":72},{"task":"/task/image-segmentation","name":"Image Segmentation","papers":67},{"task":"/task/decoder","name":"Decoder","papers":65},{"task":"/task/diagnostic","name":"Diagnostic","papers":65}],"tasks_shown":20,"n_tasks":930,"usage_by_year":[{"year":"2015","papers":6},{"year":"2016","papers":42},{"year":"2017","papers":120},{"year":"2018","papers":240},{"year":"2019","papers":380},{"year":"2020","papers":459},{"year":"2021","papers":463},{"year":"2022","papers":357},{"year":"2023","papers":368},{"year":"2024","papers":373},{"year":"2025","papers":123}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/kaiming-initialization"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}