Wiki Home

Instruction tuning

Definition

Instruction tuning fine-tunes a pretrained language model so it responds to a natural-language instruction instead of merely continuing the text. It starts from supervised fine-tuning on instruction-response demonstrations and is usually followed by preference optimization (RLHF). The result treats the prompt as a request to satisfy, which is what makes a base model into an assistant. InstructGPT (Ouyang et al., 2022) is the canonical recipe. Instruction following is one training objective among several that a deployed model balances, alongside language modeling and safety.

Papers that use this concept

• Jailbroken: How Does LLM Safety Training Fail? — instruction following is one of the “competing objectives”; attacks such as refusal suppression exploit the model’s drive to obey formatting instructions.
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep — analyzes how supervised and preference-based fine-tuning leave safety behavior concentrated in early tokens.
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — takes the instruction-tuned model as the starting point and asks what safety fine-tuning changes on top of it.
• Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — names instruction tuning as the stage that adapts a pretrained model to follow natural-language requests, the behavior an injected instruction is written to exploit.

See also

• Language model pretraining
• Reinforcement learning from human feedback (RLHF)
• Safety training (LLM harmlessness)

References

• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.