Scaling Laws for Neural Language Models
ReadLoss falls as a smooth power law in parameters, data, and compute across many orders of magnitude. Turned model sizing into something you could plan.
My notes
The plots are the paper. Straight lines on log-log axes across seven orders of magnitude is a very convincing picture.
The Claim
L(N) ≈ (N_c / N)^α_N L(D) ≈ (D_c / D)^α_D L(C) ≈ (C_c / C)^α_C
What I Kept
- Performance depends mostly on N, D and C; shape details are second order.
- Their recommendation: with more compute, grow the model much faster than the data. Chinchilla (two years later) says this was wrong — read them together.
- Bigger models are more sample-efficient: they reach the same loss with fewer tokens.