Wiki Home

A Watermark for Large Language Models

Paper: ICML 2023

High-level overview

Text produced by a large language model can be hard to tell apart from text written by a person, which complicates tracing the provenance of online content and excluding machine text from web-scale training corpora. This work embeds a watermark into generated text at decoding time so that the source of a passage can later be established from the text alone. The watermark is a hidden statistical pattern, imperceptible to a human reader yet detectable by an algorithm that holds the right key. It marks the output text rather than the model’s weights, and serves provenance, telling machine text from human text, rather than ownership. At each step the method biases the logits toward a pseudo-random “green list,” a per-position partition of the token vocabulary that shifts with the preceding context. Watermarked text then carries far more green-list tokens across a span than a human writer would produce by chance.

Detection reads the candidate text alone and applies a statistical test, yielding an interpretable p-value and a controllable false-positive rate. How strongly a passage can be watermarked depends on its entropy: high-entropy spans, where the model has many plausible next tokens, absorb the green-list bias with little change to the text, while low-entropy spans, where one continuation is nearly forced, carry almost no signal and resist watermarking. The authors derive an information-theoretic analysis relating watermark strength to next-token-distribution entropy, report that the watermark is detectable from spans as short as 25 tokens with negligible cost to text quality measured by perplexity, and study how it degrades under attacks that edit, paraphrase, or rewrite the text. Experiments use a multi-billion-parameter model from the Open Pre-trained Transformer (OPT) family (Zhang et al., 2022).

Threat Model: The watermark is embedded by the model provider, the party that controls decoding and can modify the model’s output logits before a token is drawn; embedding needs no retraining and can be retrofitted to any model that generates by sampling from a next-token distribution. The detector is any third party holding the hash function and key, for example a platform screening uploaded text; it needs only the candidate text, not the model’s parameters or API, so the detection algorithm can be open-sourced even when the model is closed, or kept private behind an API. The adversary is a user who obtains watermarked text and wants it to pass as un-watermarked: it edits tokens by insertion, deletion, or substitution, paraphrases the text by hand or with a weaker public language model, or applies generative tricks such as emoji, homoglyph, or tokenization manipulations that scramble the green-list computation; with knowledge of a public scheme it could instead try to forge the watermark onto text the provider never generated. The paper analyzes both a worst-case adversary with complete knowledge of the watermark and a setting where the scheme is kept secret, and it assumes the adversary lacks a language model as strong as the watermarked one, since an equally strong model would remove any need to use the watermarked API. The provider’s claim is that false positives are statistically improbable regardless of a human author’s writing style, and that removing the watermark from a long passage requires changing a large fraction of its tokens.

Basic Background

Language models, tokens, and decoding

A language model for next-token prediction, in practice a transformer, reads a sequence of tokens, the units of text drawn from a fixed vocabulary of tens of thousands of entries, and outputs a vector of logits, one score per vocabulary token. A softmax turns those logits into a probability distribution over the next token. Decoding is the procedure that turns these distributions into text: greedy decoding takes the most probable token, multinomial sampling draws from the distribution, and beam search keeps several high-scoring candidate continuations. The provider that runs the model sees and can alter the logits before a token is drawn, which is the hook the watermark uses.

Entropy of the next-token distribution

The entropy of a next-token distribution measures how spread out it is. A high-entropy position has many tokens of comparable probability, so several continuations are about equally good; a low-entropy position concentrates almost all mass on one token, so the next token is nearly determined by what came before. Code and stock phrases sit at the low-entropy end, where “Barack” is almost always followed by “Obama.” This quantity governs how much a sampling-time intervention can change which token is chosen without making the text worse.

Perplexity and generation quality

Perplexity is the standard intrinsic measure of how well a language model predicts a sequence, the exponential of the average per-token cross-entropy, also written as the negative log-likelihood. Lower perplexity under a reference model indicates text the model finds more fluent and predictable. It serves here as the yardstick for whether biasing the output toward green-list tokens degrades quality.

Detecting a statistical signal

Casting watermark detection as a hypothesis test treats the count of green-list tokens as a test statistic under the null hypothesis that the writer had no knowledge of the green-list rule. A one-proportion z-test compares the observed green-token fraction against the fraction expected by chance and reports a p-value, the probability of seeing so many green tokens if the null held. The trade-off between true-positive and false-positive rates as the decision threshold moves is summarized by an ROC curve and its AUC; security applications demand very low false-positive rates, since flagging human text as machine-generated is costly.

Watermarking and provenance

Digital watermarking hides an identifying signal inside a media object so its source can be recovered later; embedding hidden information into data more generally is steganography. Model watermarking marks a trained model’s weights or behavior to prove ownership of a copy. Text or output watermarking instead marks the content a model generates, so a passage can be attributed to a machine source, which is the provenance setting the green-list method targets. Watermarking discrete text has historically been harder than watermarking continuous media such as images or audio, because small edits to wording are conspicuous.

Reading guidance

• Section 3, Algorithm 2: the soft rule, which adds a constant to green-list logits so the bias yields on low-entropy text. The green list is seeded pseudo-randomly from the preceding token, which is what makes detection possible without the model.
• Section 4, Definition 4.1 and Theorem 4.2: the spike-entropy bound on expected green tokens. The theorem assumes multinomial sampling and a uniformly random green list while the deployed rule is pseudo-random; note where the text takes up that gap.
• Section 7, the sentence fixing the attacker’s resources: watermark-free text must be produced from the private watermarked model together with other public models, which the paper notes are much weaker. Every removal result in Figures 5 and 6 is relative to that grant.