Wiki Home

Jailbroken: How Does LLM Safety Training Fail?

Paper: NeurIPS 2023

High-level overview

Large language models deployed as assistants are safety-trained to refuse restricted behaviors. “Jailbreak” prompts elicit those behaviors anyway. This paper explains why safety training fails against such prompts and turns the explanation into a recipe for building attacks.

It proposes two failure modes. Competing objectives: a model is trained at once for language modeling, instruction following, and safety, and a prompt can be built that forces obeying the first two to mean violating the third. Mismatched generalization: pretraining spans a far wider range of inputs than safety training, so a prompt placed in that gap is understood by the model yet missed by its refusal behavior. Each principle yields a family of concrete attacks, and the two compose.

The paper evaluates these attacks on GPT-4, Claude v1.3, and the smaller GPT-3.5 Turbo, drawing prompts from the model creators’ own red-teaming sets and a larger held-out set. Attacks built from the two principles outperform ad hoc jailbreaks, and their combinations succeed on the large majority of restricted prompts. The paper argues these failures follow from how the models are trained rather than from their scale, so larger models alone will not remove them, and calls for “safety-capability parity,” safety mechanisms as sophisticated as the model they guard.

Threat Model: The adversary is a user of the deployed chat interface who wants to elicit a restricted behavior. Access is black-box: the adversary queries the model through the chat interface, with no view of weights or training data and no ability to set the system prompt or alter the message history. The lever is the prompt itself, rewriting a refused request into a modified prompt using transformations that are generally input-agnostic and human-readable; the attack may be adaptive, choosing the next prompt from the model’s replies to earlier variants, though most attacks succeed without adaptivity. The attack acts at inference time only, with no access to training. The defender is the model provider, which serves a safety-trained model that refuses restricted behaviors and is claimed to resist adversarial misuse.

Basic Background

Language models and pretraining

A modern LLM is first pretrained to predict the next token over a large, diverse text corpus (language model pretraining). This yields a base model whose sense of which continuations are likely, its pretraining distribution, spans far more than any later training stage revisits, including the ability to read unusual formats and encodings. Generation decodes one token at a time from this distribution, so a continuation that is improbable in natural text is also improbable for the model. How each token is actually drawn is a separate inference-time setting, the decoding strategy: a temperature at or near zero makes generation deterministic, and raising it puts the model’s less likely continuations within reach. Any success rate measured on a model is therefore a success rate at whatever decoding settings produced it.

Aligning LLMs to follow instructions

A base model continues text; an assistant follows requests. Instruction tuning supervises the model on instruction-response demonstrations, and reinforcement learning from human feedback (RLHF) then optimizes it toward human-preferred responses using a learned reward model, with a penalty that keeps it near the base model (Ouyang et al., 2022; Christiano et al., 2017). Instruction following and the residual pull of the pretraining objective are both products of this pipeline.

Safety training and refusal

Safety training is the stage that makes a deployed model refuse restricted behaviors, such as harmful content, crime facilitation, and leakage of personal data, expressed as a short refusal. It is implemented by preference optimization toward harmlessness and by AI feedback against a written policy (Constitutional AI; Bai et al., 2022), and is often paired with input and output filtering. It covers a narrower range of inputs than pretraining or instruction tuning.

Jailbreaks and red teaming

A jailbreak is a prompt engineered to elicit a restricted behavior the model would otherwise refuse; the capability is already present, and the prompt bypasses the refusal. Red teaming is the complementary practice of probing a model for policy-violating outputs to inform safety training and evaluation (Ganguli et al., 2022; Perez et al., 2022); prompts the creators red-teamed and trained against form a demanding evaluation set.

Black-box access

Black-box access means querying a model through its interface without its weights, gradients, or training data. It is the access an ordinary API or chat user has, and the setting in which these attacks operate.

Reading guidance

• Section 2.1: black-box access through a chat interface, no system-prompt modification, no edits to message history, and the GOODBOT / BADBOT / UNCLEAR labelling. The adversarial system-prompt attack in Section 4 is exempted; note where the boundary sits.
• Sections 3.1 and 3.2: competing objectives and mismatched generalization, stated as hypotheses and then used to construct attacks. Note what evidence is offered that these are the operative mechanisms.
• Figure 2 and Table 3: GPT-3.5 Turbo against GPT-4 under the same encoded prompts. Outcomes are labelled by hand; note where the two models diverge.