Adam: A Method for Stochastic Optimization

Per-parameter step sizes from running estimates of the gradient's mean and variance. The optimizer most later papers assume without discussion.

My notes

Took me two days because I stopped to re-derive bias correction by hand. Worth it — now I actually know why the first few steps are not tiny.

What Finally Clicked

m and v start at zero, so early estimates are biased toward zero. Dividing by (1 − βᵗ) undoes that; without it the first steps would be far too small.
The effective step is roughly α in magnitude regardless of gradient scale — that is why one learning rate works across very different layers.
Defaults (α=1e-3, β₁=0.9, β₂=0.999, ε=1e-8) are still what almost everyone ships.
AdamW (later paper) decouples weight decay from the adaptive step. If you are training anything transformer-shaped, you want AdamW, not Adam + L2.

Practical takeaway: when a fine-tune blows up, check the learning rate and warmup first, not the optimizer. Adam is rarely the problem.