Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov · 2014 · read 10 Sept 2026

Randomly dropping units during training makes a network behave like an averaged ensemble. A regularizer you meet in nearly every architecture.

My notes

I had used nn.Dropout(0.1) for years without reading why. The paper is long but the core fits on one page.

Mechanism

train:  h = m ⊙ f(Wx),  m ~ Bernoulli(p)
test:   h = p · f(Wx)

Think of it as training 2ⁿ networks that share weights, then averaging them cheaply at test time by scaling.
It breaks co-adaptation: a unit cannot rely on one specific neighbour being present, so features become individually useful.
Modern frameworks do "inverted dropout" — scale by 1/p during training so inference is untouched. Same thing, fewer foot-guns.
p = 0.5 for hidden layers and ~0.8 for inputs were their defaults. In transformers we use much lower rates because data is huge and overfitting is rarely the bottleneck.

Dropout is cheap ensembling hiding inside a single model.