Methods › General › Normalization › Weight Normalization
Weight Normalization
Introduced by Tim Salimans et al. in Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Weight Normalization is a normalization method for training neural networks. It is inspired by batch normalization, but it is a deterministic method that does not share batch normalization's property of adding noise to the gradients. It reparameterizes each k-dimentional weight vector w in terms of a parameter vector v and a scalar parameter g and to perform stochastic gradient descent with respect to those parameters instead. Weight vectors are expressed in terms of the new parameters using:
w = g/(‖ textbfv‖)v
where v is a k-dimensional vector, g is a scalar, and ‖v‖ denotes the Euclidean norm of v. This reparameterization has the effect of fixing the Euclidean norm of the weight vector w: we now have ‖w‖= g, independent of the parameters v.
Papers archive 2025-07-28
30 shown of 88, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Bio-Inspired Plastic Neural Networks for Zero-Shot Out-of-Distribution Generalization in Complex Animal-Inspired Robots 16 Mar 2025 · 0 repositories · arXiv:2503.12406
-
Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization 11 Feb 2025 · 0 repositories · arXiv:2502.07523
-
p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay 5 Dec 2024 · 1 repository · arXiv:2412.04449Syntology ran 8 of 8 samples · 0 unverified
-
AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach 12 Oct 2024 · 0 repositories · arXiv:2410.10896
-
Optimization and Generalization Guarantees for Weight Normalization 13 Sep 2024 · 0 repositories · arXiv:2409.08935
-
Weight Conditioning for Smooth Optimization of Neural Networks 5 Sep 2024 · 0 repositories · arXiv:2409.03424
-
Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks 24 Jul 2024 · 0 repositories · arXiv:2407.16944
-
Blood Glucose Control Via Pre-trained Counterfactual Invertible Neural Networks 23 May 2024 · 0 repositories · arXiv:2405.17458
-
PAC-Bayesian Generalization Bounds for Knowledge Graph Representation Learning 10 May 2024 · 1 repository · arXiv:2405.06418Syntology ran 3 of 4 samples · 1 unverified · 4 pointer-only (licence)
-
Hidden Synergy: L₁ Weight Normalization and 1-Path-Norm Regularization 29 Apr 2024 · 0 repositories · arXiv:2404.19112
-
XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts 23 Apr 2024 · 1 repository · arXiv:2404.15247Syntology ran 11 of 14 samples · 3 unverified
-
A2Q+: Improving Accumulator-Aware Weight Quantization 19 Jan 2024 · 1 repository · arXiv:2401.10432
-
Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities 5 Oct 2023 · 0 repositories · arXiv:2310.03696
-
Gradient-Based Feature Learning under Structured Data 7 Sep 2023 · 0 repositories · arXiv:2309.03843
-
A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance 25 Aug 2023 · 1 repository · arXiv:2308.13504
-
Robust Implicit Regularization via Weight Normalization 9 May 2023 · 0 repositories · arXiv:2305.05448
-
Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance 31 Jan 2023 · 0 repositories · arXiv:2301.13376
-
Clarinet: A Music Retrieval System 23 Oct 2022 · 1 repository · arXiv:2210.12648
-
A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck 27 Jul 2022 · 0 repositories · arXiv:2207.13529
-
WOLONet: Wave Outlooker for Efficient and High Fidelity Speech Synthesis 20 Jun 2022 · 0 repositories · arXiv:2206.09920
-
Canonical convolutional neural networks 3 Jun 2022 · 1 repository · arXiv:2206.01509
-
cMelGAN: An Efficient Conditional Generative Model Based on Mel Spectrograms 15 May 2022 · 1 repository · arXiv:2205.07319
-
Alleviating Adversarial Attacks on Variational Autoencoders with MCMC 18 Mar 2022 · 1 repository · arXiv:2203.09940
-
Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values 3 Mar 2022 · 1 repository · arXiv:2203.01993
-
Real time spectrogram inversion on mobile phone 1 Mar 2022 · 1 repository · arXiv:2203.00756
-
Deep Hybrid Models for Out-of-Distribution Detection 1 Jan 2022 · 0 repositories
-
Estimating Parameters of the Tree Root in Heterogeneous Soil Environments via Mask-Guided Multi-Polarimetric Integration Neural Network 27 Dec 2021 · 0 repositories · arXiv:2112.13494
-
Audio Deepfake Perceptions in College Going Populations 6 Dec 2021 · 0 repositories · arXiv:2112.03351
-
VocBench: A Neural Vocoder Benchmark for Speech Synthesis 6 Dec 2021 · 1 repository · arXiv:2112.03099
-
Aligned Structured Sparsity Learning for Efficient Image Super-Resolution 1 Dec 2021 · 1 repository
Tasks archive 2025-07-28
20 shown of 101 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections