Language Models are Few-Shot Learners
ReadShows that a large enough decoder can do new tasks from examples in its prompt, with no gradient updates. In-context learning and prompting start here.
My notes
75 pages, but sections 1–3 are the paper. The rest is a very honest list of what it cannot do, which I appreciated more than the results.
Notes
- Few-shot gains grow with model size. Small models barely benefit from examples; big ones use them well.
- This is the paper that made "prompt engineering" a job. It is also why evals matter — performance swings with example choice and order.
- Data contamination section is worth reading: they found benchmark overlap and tried to measure its effect. Still a live problem.
Related to my post on eval engineering: GPT-3 is the reason prompts became the main interface, and therefore the reason we need evals for them.