{"url":"/method/weight-normalization","slug":"weight-normalization","name":"Weight Normalization","full_name":"Weight Normalization","full_name_withheld":false,"description_markdown":"**Weight Normalization** is a normalization method for training neural networks. It is inspired by [batch normalization](https://paperswithcode.com/method/batch-normalization), but it is a deterministic method that does not share batch normalization's property of adding noise to the gradients. It reparameterizes each $k$-dimentional weight vector $\\textbf{w}$ in terms of a parameter vector $\\textbf{v}$ and a scalar parameter $g$ and to perform stochastic gradient descent with respect to those parameters instead. Weight vectors are expressed in terms of the new parameters using:\r\n\r\n$$ \\textbf{w} = \\frac{g}{\\Vert\\\\textbf{v}\\Vert}\\textbf{v}$$\r\n\r\nwhere $\\textbf{v}$ is a $k$-dimensional vector, $g$ is a scalar, and $\\Vert\\textbf{v}\\Vert$ denotes the Euclidean norm of $\\textbf{v}$. This reparameterization has the effect of fixing the Euclidean norm of the weight vector $\\textbf{w}$: we now have $\\Vert\\textbf{w}\\Vert = g$, independent of the parameters $\\textbf{v}$.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks","paper":"/paper/weight-normalization-a-simple","first_author":"Tim Salimans","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/weight-normalization-a-simple"},"source":{"url":"http://arxiv.org/abs/1602.07868v3","title":"Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/1c5c289b6218eb1026dcb5fd9738231401cfccea/torch/nn/utils/weight_norm.py#L8","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Normalization","url":"/methods/category/normalization","pwc_aliases":[]}],"n_papers_tagged":88,"archive_num_papers":88,"papers_newest_first":[{"paper":null,"title":"Bio-Inspired Plastic Neural Networks for Zero-Shot Out-of-Distribution Generalization in Complex Animal-Inspired Robots","date":"2025-03-16","arxiv_id":"2503.12406","n_code_links":0,"syntology":null},{"paper":null,"title":"Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization","date":"2025-02-11","arxiv_id":"2502.07523","n_code_links":0,"syntology":null},{"paper":"/paper/p-mod-building-mixture-of-depths-mllms-via","title":"p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay","date":"2024-12-05","arxiv_id":"2412.04449","n_code_links":1,"syntology":{"ran":8,"of":8,"unverified":0,"pointer_only":0}},{"paper":null,"title":"AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach","date":"2024-10-12","arxiv_id":"2410.10896","n_code_links":0,"syntology":null},{"paper":null,"title":"Optimization and Generalization Guarantees for Weight Normalization","date":"2024-09-13","arxiv_id":"2409.08935","n_code_links":0,"syntology":null},{"paper":null,"title":"Weight Conditioning for Smooth Optimization of Neural Networks","date":"2024-09-05","arxiv_id":"2409.03424","n_code_links":0,"syntology":null},{"paper":null,"title":"Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks","date":"2024-07-24","arxiv_id":"2407.16944","n_code_links":0,"syntology":null},{"paper":null,"title":"Blood Glucose Control Via Pre-trained Counterfactual Invertible Neural Networks","date":"2024-05-23","arxiv_id":"2405.17458","n_code_links":0,"syntology":null},{"paper":"/paper/pac-bayesian-generalization-bounds-for-2","title":"PAC-Bayesian Generalization Bounds for Knowledge Graph Representation Learning","date":"2024-05-10","arxiv_id":"2405.06418","n_code_links":1,"syntology":{"ran":3,"of":4,"unverified":1,"pointer_only":4}},{"paper":null,"title":"Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm Regularization","date":"2024-04-29","arxiv_id":"2404.19112","n_code_links":0,"syntology":null},{"paper":"/paper/xft-unlocking-the-power-of-code-instruction","title":"XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts","date":"2024-04-23","arxiv_id":"2404.15247","n_code_links":1,"syntology":{"ran":11,"of":14,"unverified":3,"pointer_only":0}},{"paper":"/paper/a2q-improving-accumulator-aware-weight","title":"A2Q+: Improving Accumulator-Aware Weight Quantization","date":"2024-01-19","arxiv_id":"2401.10432","n_code_links":1,"syntology":null},{"paper":null,"title":"Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities","date":"2023-10-05","arxiv_id":"2310.03696","n_code_links":0,"syntology":null},{"paper":null,"title":"Gradient-Based Feature Learning under Structured Data","date":"2023-09-07","arxiv_id":"2309.03843","n_code_links":0,"syntology":null},{"paper":"/paper/a2q-accumulator-aware-quantization-with","title":"A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance","date":"2023-08-25","arxiv_id":"2308.13504","n_code_links":1,"syntology":null},{"paper":null,"title":"Robust Implicit Regularization via Weight Normalization","date":"2023-05-09","arxiv_id":"2305.05448","n_code_links":0,"syntology":null},{"paper":null,"title":"Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance","date":"2023-01-31","arxiv_id":"2301.13376","n_code_links":0,"syntology":null},{"paper":"/paper/clarinet-a-music-retrieval-system","title":"Clarinet: A Music Retrieval System","date":"2022-10-23","arxiv_id":"2210.12648","n_code_links":1,"syntology":null},{"paper":null,"title":"A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck","date":"2022-07-27","arxiv_id":"2207.13529","n_code_links":0,"syntology":null},{"paper":null,"title":"WOLONet: Wave Outlooker for Efficient and High Fidelity Speech Synthesis","date":"2022-06-20","arxiv_id":"2206.09920","n_code_links":0,"syntology":null},{"paper":"/paper/canonical-convolutional-neural-networks","title":"Canonical convolutional neural networks","date":"2022-06-03","arxiv_id":"2206.01509","n_code_links":1,"syntology":null},{"paper":"/paper/cmelgan-an-efficient-conditional-generative","title":"cMelGAN: An Efficient Conditional Generative Model Based on Mel Spectrograms","date":"2022-05-15","arxiv_id":"2205.07319","n_code_links":1,"syntology":null},{"paper":"/paper/defending-variational-autoencoders-from","title":"Alleviating Adversarial Attacks on Variational Autoencoders with MCMC","date":"2022-03-18","arxiv_id":"2203.09940","n_code_links":1,"syntology":null},{"paper":"/paper/polarity-sampling-quality-and-diversity","title":"Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values","date":"2022-03-03","arxiv_id":"2203.01993","n_code_links":1,"syntology":null},{"paper":"/paper/real-time-spectrogram-inversion-on-mobile","title":"Real time spectrogram inversion on mobile phone","date":"2022-03-01","arxiv_id":"2203.00756","n_code_links":1,"syntology":null},{"paper":"/paper/deep-hybrid-models-for-out-of-distribution","title":"Deep Hybrid Models for Out-of-Distribution Detection","date":"2022-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Estimating Parameters of the Tree Root in Heterogeneous Soil Environments via Mask-Guided Multi-Polarimetric Integration Neural Network","date":"2021-12-27","arxiv_id":"2112.13494","n_code_links":0,"syntology":null},{"paper":null,"title":"Audio Deepfake Perceptions in College Going Populations","date":"2021-12-06","arxiv_id":"2112.03351","n_code_links":0,"syntology":null},{"paper":"/paper/vocbench-a-neural-vocoder-benchmark-for","title":"VocBench: A Neural Vocoder Benchmark for Speech Synthesis","date":"2021-12-06","arxiv_id":"2112.03099","n_code_links":1,"syntology":null},{"paper":"/paper/aligned-structured-sparsity-learning-for","title":"Aligned Structured Sparsity Learning for Efficient Image Super-Resolution","date":"2021-12-01","arxiv_id":null,"n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/speech-synthesis","name":"Speech Synthesis","papers":12},{"task":null,"name":"GPU","papers":7},{"task":"/task/image-classification","name":"Image Classification","papers":7},{"task":null,"name":"CPU","papers":6},{"task":"/task/image-generation","name":"Image Generation","papers":6},{"task":"/task/quantization","name":"Quantization","papers":6},{"task":"/task/text-to-speech","name":"Text to Speech","papers":6},{"task":"/task/text-to-speech-1","name":"text-to-speech","papers":6},{"task":"/task/decoder","name":"Decoder","papers":5},{"task":"/task/image-classification","name":"image-classification","papers":5},{"task":"/task/classification","name":"General Classification","papers":4},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":3},{"task":"/task/machine-translation","name":"Machine Translation","papers":3},{"task":"/task/model-compression","name":"Model Compression","papers":3},{"task":"/task/representation-learning","name":"Representation Learning","papers":3},{"task":"/task/translation","name":"Translation","papers":3},{"task":"/task/audio-synthesis","name":"Audio Synthesis","papers":2},{"task":"/task/classification-1","name":"Classification","papers":2},{"task":"/task/density-estimation","name":"Density Estimation","papers":2},{"task":"/task/domain-adaptation","name":"Domain Adaptation","papers":2}],"tasks_shown":20,"n_tasks":101,"usage_by_year":[{"year":"2016","papers":4},{"year":"2017","papers":7},{"year":"2018","papers":5},{"year":"2019","papers":14},{"year":"2020","papers":19},{"year":"2021","papers":13},{"year":"2022","papers":9},{"year":"2023","papers":5},{"year":"2024","papers":10},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/weight-normalization"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}