Methods › General › Normalization › Weight Normalization
Weight Normalization
Introduced by Tim Salimans et al. in Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Weight Normalization is a normalization method for training neural networks. It is inspired by batch normalization, but it is a deterministic method that does not share batch normalization's property of adding noise to the gradients. It reparameterizes each k-dimentional weight vector w in terms of a parameter vector v and a scalar parameter g and to perform stochastic gradient descent with respect to those parameters instead. Weight vectors are expressed in terms of the new parameters using:
w = g/(‖ textbfv‖)v
where v is a k-dimensional vector, g is a scalar, and ‖v‖ denotes the Euclidean norm of v. This reparameterization has the effect of fixing the Euclidean norm of the weight vector w: we now have ‖w‖= g, independent of the parameters v.
In Syntology harvested code stored, below machine verification (unverified_code; harvested), served level V1; retrieve with get_harvested_code_for_paper(arxiv_id='1602.07868'). Repositories: 9 listed of 9 in the graph, strongest link: official. The lookup matched a paper by its title, arXiv:1602.07868, not a method entry. This answer is about arXiv:1602.07868, the paper the archive credits with this method.
Origin paper of this answer, arXiv:1602.07868 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 10 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified (of 16 harvested samples) · 1 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 88 · with a code link: 45 · where Syntology ran a sample: 22 (18 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (22 of 88 tagged: 18 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 88 of 88, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Bio-Inspired Plastic Neural Networks for Zero-Shot Out-of-Distribution Generalization in Complex Animal-Inspired Robots 16 Mar 2025 · 0 repositories · arXiv:2503.12406
-
Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization 11 Feb 2025 · 0 repositories · arXiv:2502.07523
-
p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay 5 Dec 2024 · 1 repository · arXiv:2412.04449Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 7 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples) · 1 pointer-only (licence)
-
AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach 12 Oct 2024 · 0 repositories · arXiv:2410.10896
-
Optimization and Generalization Guarantees for Weight Normalization 13 Sep 2024 · 0 repositories · arXiv:2409.08935
-
Weight Conditioning for Smooth Optimization of Neural Networks 5 Sep 2024 · 0 repositories · arXiv:2409.03424
-
Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks 24 Jul 2024 · 0 repositories · arXiv:2407.16944
-
Blood Glucose Control Via Pre-trained Counterfactual Invertible Neural Networks 23 May 2024 · 0 repositories · arXiv:2405.17458
-
PAC-Bayesian Generalization Bounds for Knowledge Graph Representation Learning 10 May 2024 · 1 repository · arXiv:2405.06418Syntology official (archive's flag): 3 ran · 3 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Hidden Synergy: L₁ Weight Normalization and 1-Path-Norm Regularization 29 Apr 2024 · 0 repositories · arXiv:2404.19112
-
XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts 23 Apr 2024 · 1 repository · arXiv:2404.15247Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 9 where Syntology's instrument failed) · 3 unverified (of 14 harvested samples)
-
A2Q+: Improving Accumulator-Aware Weight Quantization 19 Jan 2024 · 1 repository · arXiv:2401.10432
-
Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities 5 Oct 2023 · 0 repositories · arXiv:2310.03696
-
Gradient-Based Feature Learning under Structured Data 7 Sep 2023 · 0 repositories · arXiv:2309.03843
-
A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance 25 Aug 2023 · 1 repository · arXiv:2308.13504
-
Robust Implicit Regularization via Weight Normalization 9 May 2023 · 0 repositories · arXiv:2305.05448
-
Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance 31 Jan 2023 · 0 repositories · arXiv:2301.13376
-
Clarinet: A Music Retrieval System 23 Oct 2022 · 1 repository · arXiv:2210.12648
-
A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck 27 Jul 2022 · 0 repositories · arXiv:2207.13529
-
WOLONet: Wave Outlooker for Efficient and High Fidelity Speech Synthesis 20 Jun 2022 · 0 repositories · arXiv:2206.09920
-
Canonical convolutional neural networks 3 Jun 2022 · 1 repository · arXiv:2206.01509
-
cMelGAN: An Efficient Conditional Generative Model Based on Mel Spectrograms 15 May 2022 · 1 repository · arXiv:2205.07319
-
Alleviating Adversarial Attacks on Variational Autoencoders with MCMC 18 Mar 2022 · 1 repository · arXiv:2203.09940
-
Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values 3 Mar 2022 · 1 repository · arXiv:2203.01993
-
Real time spectrogram inversion on mobile phone 1 Mar 2022 · 1 repository · arXiv:2203.00756
-
Deep Hybrid Models for Out-of-Distribution Detection 1 Jan 2022 · 0 repositories
-
Estimating Parameters of the Tree Root in Heterogeneous Soil Environments via Mask-Guided Multi-Polarimetric Integration Neural Network 27 Dec 2021 · 0 repositories · arXiv:2112.13494
-
Audio Deepfake Perceptions in College Going Populations 6 Dec 2021 · 0 repositories · arXiv:2112.03351
-
VocBench: A Neural Vocoder Benchmark for Speech Synthesis 6 Dec 2021 · 1 repository · arXiv:2112.03099
-
Aligned Structured Sparsity Learning for Efficient Image Super-Resolution 1 Dec 2021 · 1 repository
-
Towards Stable and Robust AdderNets 1 Dec 2021 · 0 repositories
-
Robust and Provably Monotonic Networks 30 Nov 2021 · 0 repositories · arXiv:2112.00038Syntology 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)
-
Logit Attenuating Weight Normalization 12 Aug 2021 · 0 repositories · arXiv:2108.05839
-
Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN 26 Mar 2021 · 0 repositories · arXiv:2103.14245
-
Learnable Companding Quantization for Accurate Low-bit Neural Networks 12 Mar 2021 · 0 repositories · arXiv:2103.07156
-
Towards a Unified Framework for Fair and Stable Graph Representation Learning 25 Feb 2021 · 3 repositories · arXiv:2102.13186Syntology official: not harvested · 0 ran · 1 unverified (of 1 harvested sample)
-
Neural Sampling Machine with Stochastic Synapse allows Brain-like Learning and Inference 20 Feb 2021 · 0 repositories · arXiv:2102.10477
-
Improved Training of Sparse Coding Variational Autoencoder via Weight Normalization 23 Jan 2021 · 0 repositories · arXiv:2101.09453
-
NVAE-GAN Based Approach for Unsupervised Time Series Anomaly Detection 8 Jan 2021 · 0 repositories · arXiv:2101.02908
-
Explicit regularization and implicit bias in deep network classifiers trained with the square loss 31 Dec 2020 · 0 repositories · arXiv:2101.00072
-
ParaNet: Deep Regular Representation for 3D Point Clouds 5 Dec 2020 · 0 repositories · arXiv:2012.03028
-
TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis 24 Nov 2020 · 1 repository · arXiv:2011.12206
-
Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains 19 Nov 2020 · 2 repositories · arXiv:2011.09631
-
StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalization 3 Nov 2020 · 2 repositories · arXiv:2011.01557Syntology 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Nets 24 Oct 2020 · 1 repository · arXiv:2010.12909
-
SpeedySpeech: Efficient Neural Speech Synthesis 9 Aug 2020 · 3 repositories · arXiv:2008.03802Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Learning from a Complementary-label Source Domain: Theory and Algorithms 4 Aug 2020 · 1 repository · arXiv:2008.01454Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network 30 Jul 2020 · 2 repositories · arXiv:2007.15256
-
Clarinet: A One-step Approach Towards Budget-friendly Unsupervised Domain Adaptation 29 Jul 2020 · 1 repository · arXiv:2007.14612Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
NVAE: A Deep Hierarchical Variational Autoencoder 8 Jul 2020 · 10 repositories · arXiv:2007.03898Syntology official (archive's flag): 9 ran · 26 ran (of which 15 constructed an object rather than computing a result; 21 with no instrument failure: 3 honoured, 0 violated, 18 with no contract checked; 5 where Syntology's instrument failed) · 15 unverified (of 41 harvested samples) · 23 pointer-only (licence)
-
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness 17 Jun 2020 · 4 repositories · arXiv:2006.10108Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
Adversarial representation learning for private speech generation 16 Jun 2020 · 1 repository · arXiv:2006.09114Syntology official (archive's flag): 2 ran · 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
New Interpretations of Normalization Methods in Deep Learning 16 Jun 2020 · 0 repositories · arXiv:2006.09104
-
Overparameterization and generalization error: weighted trigonometric interpolation 15 Jun 2020 · 0 repositories · arXiv:2006.08495
-
SE-MelGAN -- Speaker Agnostic Rapid Speech Enhancement 13 Jun 2020 · 0 repositories · arXiv:2006.07637
-
Optimization Theory for ReLU Neural Networks Trained with Normalization Layers 11 Jun 2020 · 0 repositories · arXiv:2006.06878
-
Single-Side Domain Generalization for Face Anti-Spoofing 29 Apr 2020 · 1 repository · arXiv:2004.14043
-
Parallel Neural Text-to-Speech 1 Jan 2020 · 0 repositories
-
Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization 29 Nov 2019 · 0 repositories · arXiv:1911.13173
-
Implicit Regularization and Convergence for Weight Normalization 18 Nov 2019 · 0 repositories · arXiv:1911.07956
-
Understanding the Disharmony between Weight Normalization Family and Weight Decay: ε-shifted L₂ Regularizer 14 Nov 2019 · 1 repository · arXiv:1911.05920
-
Systematic Comparison of the Influence of Different Data Preprocessing Methods on the Performance of Gait Classifications Using Machine Learning 11 Nov 2019 · 0 repositories · arXiv:1911.04335
-
Inherent Weight Normalization in Stochastic Neural Networks 27 Oct 2019 · 1 repository · arXiv:1910.12316
-
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis 8 Oct 2019 · 21 repositories · arXiv:1910.06711Syntology official (archive's flag): 1 ran · 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 1 pointer-only (licence)
-
Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks 28 Sep 2019 · 1 repository · arXiv:1909.13144
-
Adversarial Lipschitz Regularization 12 Jul 2019 · 2 repositories · arXiv:1907.05681Syntology 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Multi-Speaker End-to-End Speech Synthesis 9 Jul 2019 · 0 repositories · arXiv:1907.04462
-
Weight Normalization based Quantization for Deep Neural Network Compression 1 Jul 2019 · 0 repositories · arXiv:1907.00593
-
How to Initialize your Network? Robust Initialization for WeightNorm & ResNets 5 Jun 2019 · 2 repositories · arXiv:1906.02341Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 10 harvested samples) · 10 pointer-only (licence)
-
Graph Neural Networks Exponentially Lose Expressive Power for Node Classification 27 May 2019 · 1 repository · arXiv:1905.10947Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Non-Autoregressive Neural Text-to-Speech 21 May 2019 · 2 repositories · arXiv:1905.08459Syntology 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)
-
Neural source-filter waveform models for statistical parametric speech synthesis 27 Apr 2019 · 0 repositories · arXiv:1904.12088
-
WaveGlow: A Flow-based Generative Network for Speech Synthesis 31 Oct 2018 · 2 repositories · arXiv:1811.00002Syntology 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples) · 2 pointer-only (licence)
-
Fine-tuning on Clean Data for End-to-End Speech Translation: FBK @ IWSLT 2018 16 Oct 2018 · 0 repositories · arXiv:1810.07652
-
The RWTH Aachen University English-German and German-English Unsupervised Neural Machine Translation Systems for WMT 2018 1 Oct 2018 · 0 repositories
-
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech 19 Jul 2018 · 5 repositories · arXiv:1807.07281Syntology 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 1 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 13 harvested samples) · 1 pointer-only (licence)
-
Orthogonal Weight Normalization: Solution to Optimization overMultiple Dependent Stiefel Manifolds in Deep Neural Networks 2 Feb 2018 · 1 repository
-
Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima 3 Dec 2017 · 0 repositories · arXiv:1712.00779
-
Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning 20 Oct 2017 · 7 repositories · arXiv:1710.07654Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Projection Based Weight Normalization for Deep Neural Networks 6 Oct 2017 · 1 repository · arXiv:1710.02338
-
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification 24 Sep 2017 · 0 repositories · arXiv:1709.08145
-
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks 16 Sep 2017 · 1 repository · arXiv:1709.06079
-
Churn Identification in Microblogs using Convolutional Neural Networks with Structured Logical Knowledge 1 Sep 2017 · 0 repositories
-
On the Effects of Batch and Weight Normalization in Generative Adversarial Networks 13 Apr 2017 · 3 repositories · arXiv:1704.03971
-
Multiplicative LSTM for sequence modelling 26 Sep 2016 · 1 repository · arXiv:1609.07959
-
Improved Techniques for Training GANs 10 Jun 2016 · 46 repositories · arXiv:1606.03498Syntology community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
Density estimation using Real NVP 27 May 2016 · 35 repositories · arXiv:1605.08803Syntology 48 ran (of which 26 constructed an object rather than computing a result; 38 with no instrument failure: 2 honoured, 0 violated, 36 with no contract checked; 10 where Syntology's instrument failed) · 22 unverified (of 70 harvested samples) · 33 pointer-only (licence)
-
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks 25 Feb 2016 · 9 repositories · arXiv:1602.07868Syntology community repositories only · 10 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified (of 16 harvested samples) · 1 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 101 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections