Distilling the Knowledge in a Neural Network
ReadTrains a small model to match a large model's softened output distribution rather than hard labels. The basis for most model compression and teacher-student setups.
My notes
Directly useful at work: we distil bigger-model judgements into cheaper models all the time. The "dark knowledge" framing is the part I will keep repeating.
Teacher → Student
q_i = exp(z_i / T) / Σ_j exp(z_j / T)
Why Soft Targets Work
- A hard label says "this is a 2". The teacher's softmax says "this is a 2, looks a bit like a 7, nothing like a 4". That similarity structure is the dark knowledge.
- Scale the soft loss by T² so its gradients stay comparable to the hard-label loss.
- Same idea shows up everywhere now: distilling LLMs into small models, training rerankers from LLM scores, synthetic-data fine-tunes.