Direct Preference Optimization: Your Language Model is Secretly a Reward Model
ReadRewrites the RLHF objective so the policy trains directly on preference pairs, with no reward model or sampling loop. Read it to see what the RL step was actually computing.
My notes
Last paper of the stretch and a satisfying one to end on: it takes the InstructGPT pipeline and collapses two of its three steps into a single classification-style loss.
RLHF vs DPO
L_DPO = −E[ log σ( β·log(π_θ(y_w|x)/π_ref(y_w|x)) − β·log(π_θ(y_l|x)/π_ref(y_l|x)) ) ]
Notes
- y_w = chosen response, y_l = rejected. The loss pushes up the log-ratio of the chosen one relative to the reference model, and down for the rejected.
- No sampling during training, no separate reward model, far fewer hyperparameters. Much easier to run on a small budget.
- β plays the same role as the KL coefficient in PPO — how far you let the policy drift from the reference.
If I fine-tune a reviewer model on "good comment vs noisy comment" pairs, this is the first thing I would try.