Methods › General › Normalization › Online Normalization

Online Normalization

3 papers tagged archive 2025-07-28

Introduced by Vitaliy Chiley et al. in Online Normalization for Training Neural Networks

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

Online Normalization is a normalization technique for training deep neural networks. To define Online Normalization. we replace arithmetic averages over the full dataset in with exponentially decaying averages of online samples. The decay factors α_f and α_b for forward and backward passes respectively are hyperparameters for the technique.

We allow incoming samples xₜ, such as images, to have multiple scalar components and denote feature-wide mean and variance by μ(xₜ) and σ²(xₜ). The algorithm also applies to outputs of fully connected layers with only one scalar output per feature. In fact, this case simplifies to μ(xₜ) = xₜ and σ(xₜ) = 0. Denote scalars μₜ and σₜ to denote running estimates of mean and variance across all samples. The subscript t denotes time steps corresponding to processing new incoming samples.

Online Normalization uses an ongoing process during the forward pass to estimate activation means and variances. It implements the standard online computation of mean and variance generalized to processing multi-value samples and exponential averaging of sample statistics. The resulting estimates directly lead to an affine normalization transform.

yₜ = (xₜ - μₜ₋₁)/(σₜ₋₁)

μₜ = α_fμₜ₋₁ + (1-α_f)μ(xₜ)

σ²ₜ = α_fσ²ₜ₋₁ + (1-α_f)σ²(xₜ) + α_f(1-α_f)(μ(xₜ) - μₜ₋₁)²

Paper Source

Papers archive 2025-07-28

3 shown of 3, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.

One model to enhance them all: array geometry agnostic multi-channel personalized speech enhancement
20 Oct 2021 · 0 repositories · arXiv:2110.10330
Pipelined Backpropagation at Scale: Training Large Models without Batches
25 Mar 2020 · 0 repositories · arXiv:2003.11666
Online Normalization for Training Neural Networks
15 May 2019 · 1 repository · arXiv:1905.05894

Syntology ran 0 of 16 samples · 16 unverified

Tasks archive 2025-07-28

14 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

Task	Papers
Image Classification	2
All	1
Automatic Speech Recognition	1
Automatic Speech Recognition (ASR)	1
General Classification	1
Image Segmentation	1
Language Modeling	1
Language Modelling	1
Semantic Segmentation	1
Speech Enhancement	1
Speech Recognition	1
Stochastic Optimization	1
image-classification	1
speech-recognition	1

Usage over time archive 2025-07-28

Papers per year the archive tags with this method, by the paper's archive date (3 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Normalization

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections