Wiki Home

Safety training (LLM harmlessness)

Definition

Safety training is the stage that makes a deployed language model refuse a designated set of restricted behaviors, such as producing harmful content, abetting crime, or leaking personally identifiable information. The trained response is a refusal: a short, characteristic decline. It is implemented by preference optimization toward harmlessness (RLHF) and by AI feedback against a written policy (Constitutional AI; Bai et al., 2022), and is often paired with input and output filtering. Safety training covers a narrower range of inputs than pretraining or instruction tuning, which is why a deployed model can retain a capability while refusing to exercise it on the prompts safety training anticipated.

Papers that use this concept

• Jailbroken: How Does LLM Safety Training Fail? — the paper’s subject; it asks why safety training fails to suppress restricted behavior under adversarial prompts.
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep — argues current safety training is “shallow,” changing mostly the first few output tokens, and proposes ways to make it deeper.
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — gives a mechanistic account of what safety training changes: a minimal MLP-weight transformation that clusters unsafe inputs apart from safe ones.
• Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — names safety training as the later stage that trains the deployed model to refuse a set of restricted behaviors, the refusal a jailbreak-style payload targets.

See also

• Reinforcement learning from human feedback (RLHF)
• Jailbreak (LLM)
• Red teaming (language models)

References

• Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. “Constitutional AI: Harmlessness from AI Feedback.” arXiv:2212.08073, 2022.
• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.