Wiki Home

Extracting Training Data from Large Language Models

Paper: USENIX Security 2021

High-level overview

A language model trained by next-token prediction assigns high probability to its training data, so some memorization of that data is intrinsic to the objective. This work asks whether a large model trained roughly once over a massive, de-duplicated corpus, a regime with almost no gap between training and test loss, still memorizes individual examples tightly enough that an outsider can pull them back out word for word. It answers yes, with a training-data extraction attack that pairs broad sampling of candidate continuations with a membership-inference ranking signal: a sequence the target model finds far less surprising than a reference model does is flagged as likely verbatim training text.

Run against GPT-2, a model trained on text scraped from the public web, the attack recovers hundreds of verbatim training sequences, including personally identifiable information (names, phone and fax numbers, email and physical addresses), source code, news headlines, and high-entropy strings such as UUIDs, many present in only a single training document. The paper pins down its target as k-eidetic memorization: a string the model reproduces that appears in at most k of the training documents, with small k the dangerous regime, where “eidetic” borrows the term for recall after a single exposure. It further reports that within the GPT-2 family, larger models memorize more.

Threat Model: The adversary has black-box query access to a trained, deployed language model. It can submit a prefix, sample continuations, and read the probability the model assigns to any sequence, but cannot inspect weights, gradients, or hidden states. It acts only at inference time on an already-trained model and changes nothing about training. It does not target a particular person or record; the goal is to extract memorized training data indiscriminately, prioritizing verbatim sequences (distinct from the fuzzy, representative reconstructions of model inversion) that appear in few training documents. Because memorized content surfaces through ordinary, honest generation, the “adversary” can be an algorithm or even a benign user, and the party at risk is whoever trained the model on a corpus that may contain private text.

Basic Background

Language models and text generation

A neural language model is trained by next-token prediction: it estimates the probability of each token given the preceding ones, and the training loss is the average cross-entropy, equivalently the negative log-likelihood, over the corpus. The same model generates text autoregressively, sampling one token at a time and feeding it back; the decoding strategy (greedy selection, temperature scaling, or top-k sampling) controls how each next token is drawn from the predicted distribution. The standard intrinsic quality measure is perplexity, the exponential of the per-token average negative log-likelihood, low when the model finds a sequence unsurprising. GPT-2 (Radford et al., 2019) is a Transformer language model of this kind (Vaswani et al., 2017), released in sizes from 124 million to 1.5 billion parameters and trained on text scraped from the public web.

Memorization and overfitting

A model memorizes an example when its behavior on that example depends strongly on the example being in the training set. Memorization is often conflated with overfitting, the gap between low training error and higher test error, but the two come apart: the generalization gap is an average over the data, while memorization is per-example and can persist in a model whose average training and test losses are nearly identical. Theory contemporaneous with this paper argues that fitting the long tail of rare examples may be necessary for accuracy, so well-generalizing models memorize some data by design (Feldman, 2020; Feldman and Zhang, 2020).

Training-data privacy attacks

Privacy attacks recover information about a model’s training set from access to the model. The least revealing is membership inference, which decides whether a given example was in the training set (Shokri et al., 2017); its baseline thresholds the model’s confidence, since models tend to be more confident on training data, and its success was tied to overfitting (Yeom et al., 2018). Model inversion instead reconstructs a representative, often fuzzy, view of a class (Fredrikson et al., 2015). Training-data extraction is stronger still: it aims to reproduce specific training examples verbatim. Sharper membership tests compare the target model’s score against a second reference model, a likelihood-ratio style calibration that down-weights sequences any model would rate as likely. Every such attack assumes a particular access level, from white-box weights and gradients to query-only black-box access.

Differential privacy as a defense

The standard principled defense is to train with differential privacy, which bounds how much any single training record can affect the model. DP-SGD is its deep-learning instantiation (Abadi et al., 2016), and it has been applied to recurrent language models (McMahan et al., 2018). The guarantee comes at a cost in accuracy and training time, and that cost falls hardest on the long-tail examples that contribute most to quality.

Reading guidance

• Section 3, Definitions 1 and 2: what makes a string extractable, and k-eidetic memorization, defined by counting the training documents containing it. Definition 1 admits pathological cases; note what footnote 4 lists and how short prefixes sidestep them.
• Section 5.2, with the metric list in Section 6.1: candidates are ranked by five comparison signals (Small and Medium GPT-2, zlib entropy, lower-cased text, a 50-token sliding window) set against a plain-perplexity baseline. The comparison, not the perplexity, is the attack.
• Section 6.2 with Table 2: 604 of 1,800 candidates confirmed memorized, an aggregate true-positive rate of 33.5%, broken out by generation strategy and inference metric. Section 6.1 fixes what confirmation meant, one of four authors finding a non-trivial substring that returns an exact Google match; note what that makes the 604 a count of.