Knowledge note

Layer Normalization

A normalization method that standardizes each example or token across its feature dimensions, independently of other batch members.

Updated September 16, 2026Confidence: Medium

Layer Normalization

Layer Normalization is a normalization method that standardizes one example or token across its feature dimensions, independently of other batch members.

Why It Exists

Sequence models process variable lengths and may train or infer with very different batch compositions. Batch-dependent statistics can therefore be noisy or mismatched, while LayerNorm gives each token a stable, local normalization rule.

How It Works

For a hidden vector $x$ of width $D$, LayerNorm computes its population mean $\mu$ and variance $\sigma^2$, then returns

\[y=\gamma\odot\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta,\]

where $\gamma$ and $\beta$ are learned featurewise parameters. In a Transformer tensor shaped [batch, sequence, hidden], the reduction is normally over hidden only.

Tradeoffs

LayerNorm is independent of batch size and needs no running statistics, but it discards the token’s overall feature mean and scale before the learned affine transform. It can also be less suitable than BatchNorm when population-level channel statistics are useful.

Common Confusions

  • LayerNorm vs. BatchNorm: LayerNorm reduces features within one token or example; BatchNorm reduces examples for each feature or channel.
  • Statistics vs. parameters: Mean and variance are recomputed from the current input; only $\gamma$ and $\beta$ are learned.
  • Normalization vs. normal distribution: Standardization gives zero mean and unit variance before the affine transform; it does not make the values Gaussian.

Where It Appears