NOTES / FINE-TUNING AND ADAPTATION
Preference tuning: RLHF and DPO
How models are tuned from comparisons rather than single right answers: the RLHF pipeline, DPO's shortcut, and reinforcement learning with verifiable rewards.
Why preferences
For many tasks there is no single correct answer, but people can say which of two answers is better. Preference tuning uses those comparisons. It sits after supervised fine-tuning and is how most chat models were made helpful and harmless.
RLHF
InstructGPT set the pattern. First, supervised fine-tuning on labeler demonstrations. Second, collect rankings of model outputs and train a reward model on them. Third, fine-tune the model with reinforcement learning to maximise that reward1. The three-step recipe was reported to make a 1.3B InstructGPT model preferred over the 175B GPT-3 in human evaluations, and to roughly halve a measured hallucination rate (21 percent versus 41 percent on their closed-domain tasks)1.
Diagram: my own.
The reinforcement-learning step typically adds a penalty for drifting too far from the starting model, so that the policy does not exploit the reward model. RLHF works but is complex: it needs a reward model, sampling during training, and careful tuning.
DPO
Direct Preference Optimization removes the separate reward model and the reinforcement-learning loop. Its authors show that the constrained reward-maximisation problem can be optimised in a single stage that amounts to a classification loss on preference pairs, and describe the result as stable, performant and lightweight2.
Diagram: my own.
Practical notes (my own, from using it): DPO needs pairs of chosen and rejected responses, ideally from the model’s own distribution; it is easy to run with adapters; and the quality of the pairs matters more than the optimiser. Several variants exist; check the library you use.
Verifiable rewards
Some tasks have answers a program can check: maths with a known result, code with tests, formats with a schema. Reinforcement learning with verifiable rewards (RLVR) replaces the learned reward model with that checker. Ai2’s Tülu 3 post-training recipe included an RLVR stage, and the team reports that it improved maths performance more at larger scale3. DeepSeek-R1 trained its reasoning behaviour using rule-based rewards with the GRPO algorithm, and its authors cite susceptibility of neural reward models to reward hacking as a reason45. GRPO shares much with PPO6.
Diagram: my own.
Limits: it only works where answers can be verified, and a weak checker can be gamed. For agents, tests and tool outcomes can act as checkers, which is an active research area (Coding agents).
What this means for builders
- With hosted APIs, preference tuning is the provider’s concern. Where offered, you may get DPO or reinforcement fine-tuning options; check current docs.
- If you self-host, start with SFT. Add DPO only when you have a clear preference signal, such as logged accepted versus edited outputs, and an eval to confirm it helped.
- Preference data encodes human judgement, including its biases. Audit it (Preparing training data) and test for regressions (Red teaming).
- Do not expect preference tuning to add knowledge (When to fine-tune).
Sources
- [1]Training language models to follow instructions with human feedback — Ouyang et al. · 2022PARAPHRASED
- [2]Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al. · 2023PARAPHRASED
- [3]Scaling the Tülu 3 post-training recipes to surpass the performance of DeepSeek V3 — Ai2 · 2025PARAPHRASED
- [4]Reinforcement Learning with Verifiable Rewards (RLVR) — howaiworks.ai glossary · 2026DESCRIPTION OF RLVR AND THE REWARD-HACKING CONCERN, AS SUMMARISED BY A THIRD PARTY
- [5]Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification — 2025BACKGROUND ON GRPO IN THE DEEPSEEK MODELS
- [6]Reinforcement learning with verifiable reward — Fireworks AIRELATIONSHIP BETWEEN GRPO AND PPO, AS SUMMARISED BY A VENDOR