Training Language Models to Follow Instructions with Human Feedback
ReadApplies supervised fine-tuning, a preference reward model, and PPO to make GPT-3 follow instructions. The alignment pipeline every chat model descends from.
My notes
The headline result still surprises me: labelers preferred the 1.3B InstructGPT over the 175B GPT-3. Alignment beat 100× parameters.
Three Steps
objective = E[ r_θ(x, y) ] − β · KL( π_RL ‖ π_SFT )
What I Noted
- Rankings are easier and more consistent for humans than writing ideal answers — that is why the reward model is trained on comparisons.
- The "alignment tax": RLHF hurt some benchmark scores; mixing pre-training gradients back in (PPO-ptx) reduced it.
- Labeler instructions and demographics are in the appendix. The model aligns to those people, not to "humans" in general.