Training Compute-Optimal Large Language Models
ReadRevisits the scaling fits and finds most large models were undertrained on data for their size. Explains why later models use far more tokens per parameter.
My notes
Read it the same day as Kaplan et al. The disagreement between the two is the most educational part of the scaling literature.
Takeaways
- Kaplan said grow N faster than D. Chinchilla says grow them in equal proportion. Difference comes partly from learning-rate schedules not being tuned to run length.
- Smaller-but-longer-trained models are also cheaper at inference — that is the part that matters most for people deploying models.
- Modern open models train far beyond 20 tokens/param on purpose, because inference cost dominates over their lifetime.