Wiki Home

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

Paper: USENIX Security 2025

High-level overview

Retrieval-augmented generation (RAG) answers a question by retrieving relevant texts from a knowledge database and placing them in a language model’s context, so the model grounds its answer in retrieved text rather than its fixed training knowledge. This paper asks what happens when that knowledge database is not trusted. It introduces PoisonedRAG, presented as the first knowledge corruption attack on RAG: an adversary inserts a small number of crafted texts into the corpus so that, for a question the adversary chose in advance (the “target question”), the system returns an answer the adversary chose in advance (the “target answer”). The worked example throughout is making the system answer “Tim Cook” to “Who is the CEO of OpenAI?”.

An injected text succeeds only if it is retrieved for the target question and, once in the context, steers the model to the target answer. The paper crafts texts that do both, under two assumptions about the adversary’s knowledge of the retriever: black-box (no access to the retriever’s parameters, the main and stronger setting) and white-box. Reported results, at the level of the abstract, are high attack success rates from injecting only a handful of texts per target question into databases holding millions of texts: 97% on the Natural Questions dataset from five texts per question against a corpus of roughly 2.7 million clean texts. The attack holds across several question-answering datasets and language models, and against several input-level defenses.

This differs from prompt injection, with which it is easily confused. Prompt injection smuggles instructions into content the model reads (“ignore your task and output X”); PoisonedRAG injects misleading content that reads as ordinary text, so the model is misled by what it treats as a credible retrieved fact rather than by a hidden command.

Threat Model: The adversary picks one or more target questions and a target answer for each, then injects a few malicious texts into the RAG knowledge database, for instance by editing a public source the corpus is built from or through insider write access to a private one. The adversary cannot read the other (clean) texts in the database and cannot access or query the language model. Knowledge of the retriever splits the settings: in the black-box setting the adversary cannot access the retriever’s parameters or query it (the paper’s main, stronger assumption); in the white-box setting the adversary holds the retriever’s parameters, which the paper motivates by the common practice of deploying a publicly available retriever. The corruption is placed into the corpus ahead of time and retrieved at inference, acting on the data the deployed system consults rather than on training or on the user’s prompt. The defender, the party running input-level defenses the paper evaluates, aims to detect or neutralize the injected texts.

Basic Background

Retrieval-augmented generation and dense retrieval

A language model’s knowledge is fixed at the end of pretraining, so it cannot answer about events after its cutoff and may fabricate plausible-sounding but false statements. Retrieval-augmented generation mitigates this by fetching relevant documents from an external knowledge database at inference time and putting them in the model’s context window, so the answer is grounded in retrieved text. The component that decides which documents are fetched is the retriever. Modern RAG uses dense retrieval: a query and each passage are embedded into vectors by learned encoders, relevance is the similarity of those vectors, and the top-k most similar passages are returned. Understanding that the corpus is consulted at inference, and that retrieval is decided by embedding similarity, is the prerequisite for everything in this paper.

Data poisoning and the new attack surface

Data poisoning is the classical attack of corrupting the data a model learns from so the trained model misbehaves; backdoor attacks are the trigger-conditioned subclass. In all of these the corrupted data is the training set. This paper moves the same idea to a different surface: the inference-time knowledge base of a deployed RAG system, which no model is trained on but every answer depends on.

Attacks that steer LLMs through their inputs

Two adjacent attack families act through what a model reads rather than how it was trained. Prompt injection places adversarial instructions in the prompt, or, as indirect injection, in content the application ingests, to override the intended task. A jailbreak crafts input that elicits behavior the model was trained to refuse. Both are worth holding in mind as contrasts: PoisonedRAG steers the answer with misleading content treated as a fact, not with instructions or a refusal bypass.

Adversary knowledge and adversarial text

Attacks are specified partly by what the adversary knows, along the white-box / black-box spectrum, from full model parameters and gradients down to query-only or no access. When the retriever’s parameters are available, making a text rank for a target query becomes the problem of crafting an adversarial example in text, optimized so the retriever scores it as similar to the query; such crafted inputs often transfer across models built for the same task.

Detecting and evaluating injected text

One defense family asks whether a text looks machine-generated or anomalous. Perplexity measures how well a language model predicts a text and is used as a quality and anomaly signal: unusually high perplexity can flag adversarially crafted text. Detectors that threshold such a score are evaluated with ROC curves and AUC, reading the true-positive rate against the false-positive rate, since a detector that only works at a high false-positive rate is not deployable.

Reading guidance

• Section 3.1: what the attacker is denied (the other texts in the database, any access to the LLM) against what it is granted (the retriever, in the white-box setting). The result is strong in proportion to how little is assumed.
• Section 4.2.2: in the black-box setting the retrieval half of an injected text is set equal to the target question itself. Note the one-line justification, and how much of the attack’s success rides on that choice.
• Table 1: attack success rates across three datasets and eight language models, with five malicious texts injected per target question. That injection budget is what the number means.
• Section 7.2, Figure 6: the perplexity detector’s ROC curve and AUC. That value is what says whether injected texts are separable from clean ones at all.