Wiki Home

Reinforcement learning from human feedback (RLHF)

Definition

RLHF aligns a language model with human preferences. Annotators rank model outputs, a reward model is trained to predict those rankings, and the policy (the language model) is then optimized to maximize predicted reward (Christiano et al., 2017; Ouyang et al., 2022). The optimization carries a KL-divergence penalty that keeps the policy close to the base model and often a term retaining the pretraining loss. RLHF underlies both instruction following and safety behavior in models such as InstructGPT and the helpful-and-harmless assistant (Bai et al., 2022). Because the base model and pretraining objective stay in the loss, alignment adjusts the base model rather than replacing it.

Papers that use this concept

• Jailbroken: How Does LLM Safety Training Fail? — the persistence of the pretraining objective inside the RLHF loss is central to its argument that scaling will not remove “competing objectives.”
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep — uses the KL-penalty view of alignment to argue that current safety alignment concentrates in the first few output tokens.
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — situates RLHF among the safety fine-tuning objectives it studies, alongside supervised fine-tuning, DPO, and unlearning.
• Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — names RLHF alongside instruction tuning as the adaptation that makes the deployed model follow requests.

See also

• Instruction tuning
• Safety training (LLM harmlessness)
• Language model pretraining

References

• Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. “Deep Reinforcement Learning from Human Preferences.” Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
• Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.” arXiv:2204.05862, 2022.