Wiki Home
Jailbreak (LLM)
Definition
A jailbreak is a prompt crafted to make a safety-trained language model produce a restricted behavior it was trained to refuse, such as harmful instructions or leaked personal data. The model already holds the underlying capability; the jailbreak routes around the refusal rather than teaching the model anything new. A jailbreak differs from an adversarial example, which perturbs an input to cause a misclassification: a jailbreak elicits an unsafe capability, and the prompts are usually human-readable and input-agnostic. Early jailbreaks spread informally after ChatGPT’s release, including role-play personas such as “DAN” (walkerspider, 2022; Burgess, 2023).
Papers that use this concept
- Jailbroken: How Does LLM Safety Training Fail? — proposes two failure modes of safety training and uses them as principles for constructing jailbreaks.
- Safety Alignment Should Be Made More Than Just a Few Tokens Deep — explains prefilling, adversarial-suffix, and decoding-parameter jailbreaks as ways of getting past a shallow refusal prefix.
- PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models — names jailbreaking as orthogonal: it steers a RAG answer through corrupted retrieved content rather than by bypassing a safety refusal.
- What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — models jailbreaks as competing objectives and mismatched generalization, and shows they evade the safety mechanism by producing activations that resemble safe inputs.
- Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — holds jailbreaking apart from prompt injection as the related move of eliciting a behavior the model was safety-trained to refuse, and uses jailbreak-style payloads as one kind of indirect injection.
See also
References
- walkerspider. “DAN is my new friend.” Reddit r/ChatGPT, 2022.
- Burgess, M. “The Hacking of ChatGPT is Just Getting Started.” Wired, 2023.