Wiki Home
White-box and black-box access
Definition
Access assumptions state what the adversary knows about the target model. A white-box adversary has everything: architecture, weights, and therefore gradients. A black-box adversary sees only the model’s outputs through queries, or has no query access at all (sometimes called a “no-box” adversary) and must rely on transferability from a surrogate model. Between the two sit intermediate notions; one is the first-order adversary, which may compute gradients of the loss with respect to the input but use no higher-order information. White-box is the worst case for the defender, so defense evaluations default to it. Papers define “grey-box” inconsistently, so read each paper’s definition rather than assuming one.
Papers that use this concept
- Towards Deep Learning Models Resistant to Adversarial Attacks — evaluates white-box and transfer adversaries and argues its guarantee for the first-order class.
- Membership Inference Attacks From First Principles — its adversary is black-box, querying confidence outputs with no access to weights or gradients.
- Deep Learning with Differential Privacy — its guarantee holds against a white-box adversary with full access to the released parameters.
- Jailbroken: How Does LLM Safety Training Fail? — its adversary is a black-box chat user who rewrites the prompt but cannot see weights or alter the system prompt or message history.
- Extracting Training Data from Large Language Models — its adversary has only black-box query access: it samples continuations and reads sequence probabilities, with no view of weights or hidden states.
- Knockoff Nets: Stealing Functionality of Black-Box Models — its adversary has only black-box query access to an image classifier, reading posterior probability vectors with no view of weights, architecture, or training data.
- DAWN: Dynamic Adversarial Watermarking of Neural Networks — both the extraction adversary and the owner’s ownership check operate over black-box query access to a prediction API.
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks — the defender has white-box access to the trained model (weights, gradients, neuron activations) but not to its training data.
- PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models — splits the attack by the adversary’s knowledge of the retriever: black-box (no parameters, no queries) versus white-box (retriever parameters available to optimize against).
- What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — its mechanistic analysis is white-box throughout, reading activations, weights, and gradients at every layer, and its embedding attack assumes white-box access.
- SoK: Unintended Interactions among Machine Learning Defenses and Risks — spans defenses and risks across both white-box and black-box access to the model, defining its observables over that range.
- Examining Zero-Shot Vulnerability Repair with Large Language Models — the developer or repair pipeline has black-box access to the commercial code models, seeing only generated completions through an API, not weights or gradients.
- No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML — the protection goal is downgrading white-box stealing and membership inference to black-box label-only attacks; the paper measures how far short the partitions fall.
- ModelGuard: Information-Theoretic Defense Against Model Extraction Attacks — its adversary has black-box query access to the prediction API, reading perturbed confidence vectors, and in the strong case additionally knows the target’s architecture.
- Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — its adversary has black-box access and need not query the model at all, planting a payload in content the victim’s model will later retrieve.