Wiki Home

Red teaming (language models)

Definition

Red teaming deliberately probes a model for outputs that violate its safety policy, both to surface weaknesses before deployment and to generate data the model is then trained to refuse. The probing prompts may be written by humans or generated by another model (Perez et al., 2022; Ganguli et al., 2022). Red-teaming datasets double as evaluation benchmarks: prompts the model creators judged harmful and trained against are a stringent test, since any attack that still succeeds on them has defeated the cases safety training targeted directly.

Papers that use this concept

• Jailbroken: How Does LLM Safety Training Fail? — its curated evaluation prompts are drawn from the OpenAI and Anthropic red-teaming sets.

See also

• Safety training (LLM harmlessness)
• Jailbreak (LLM)

References

• Perez, E., Huang, S., Song, H. F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. “Red Teaming Language Models with Language Models.” Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
• Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned.” arXiv:2209.07858, 2022.