Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
ReadNormalizing activations per mini-batch lets much deeper networks train with larger learning rates. LayerNorm and RMSNorm are variations on this move.
My notes
The "internal covariate shift" explanation has since been questioned, but the technique clearly works. Good reminder that the reason a paper gives and the reason a method works can differ.
Things I Had Wrong Before Reading
- At inference it uses running averages of μ and σ², not the batch. Forgetting model.eval() is the classic bug.
- It behaves badly with tiny batches — which is exactly why transformers use LayerNorm (normalize over features, not the batch).
- RMSNorm, used in most current LLMs, drops the mean-centering and β entirely. Cheaper and apparently just as good.