Wiki Home

Direct preference optimization (DPO)

Definition

Direct preference optimization aligns a language model to pairwise human preferences, a preferred response versus a dispreferred one, without training a separate reward model or running a reinforcement-learning loop (Rafailov et al., 2023). It rewrites the KL-regularized reward-maximization objective behind RLHF so that the optimal policy can be fit directly with a binary classification loss over preference pairs. The model’s own log-probabilities, measured relative to a frozen reference model, stand in for an implicit reward, and a KL term keeps the policy close to that reference.

DPO targets the same goal as RLHF, a policy that favors human-preferred outputs while staying near a reference model, but reaches it with ordinary supervised optimization rather than online sampling and reward estimation. RLHF and DPO are the two dominant preference-optimization recipes behind instruction following and safety behavior in deployed models.

Papers that use this concept

• Safety Alignment Should Be Made More Than Just a Few Tokens Deep — names DPO as part of the standard alignment stack; its KL-regularized preference formulation is one basis for the paper’s token-wise constrained fine-tuning objective.
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — DPO is one of the three safety fine-tuning protocols it compares, and one of the stronger ones by cluster separation and reduced sensitivity.

See also

• Reinforcement learning from human feedback (RLHF)
• Instruction tuning
• Safety training (LLM harmlessness)

References

• Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” Advances in Neural Information Processing Systems (NeurIPS), 36, 2023.