Wiki Home

Knockoff Nets: Stealing Functionality of Black-Box Models

Paper: CVPR 2019

High-level overview

A deployed image classifier exposed through a prediction API answers queries, image in and class probabilities out, while keeping its training data, its architecture, and even the meaning of its output classes private. This work asks how much of such a model’s value an adversary can capture from that interface alone. It formulates functionality stealing: training a substitute model, the “knockoff,” that matches the victim’s accuracy on the victim’s own task, without recovering its weights and without reproducing it prediction for prediction. The attack builds a “transfer set” of image-prediction pairs by querying the victim, then trains the knockoff on that set to imitate the victim’s outputs. Query images come from a large public image collection that need not overlap the victim’s training distribution, chosen either by a random strategy or by an adaptive strategy that casts query selection as a reinforcement-learning problem for better sample-efficiency.

The reported observations: a knockoff trained on images from a different distribution still recovers a large fraction of the victim’s accuracy (on the order of 0.81 to 0.96 times the victim’s test accuracy across several datasets in the hardest “open-world” setting, where the query pool and the victim’s classes barely overlap); this holds even when the knockoff uses a different architecture than the victim; and the attack carries over to a commercial image-analysis API, where a usable knockoff was trained for about $30 in queries.

Threat Model: The adversary has black-box query access to a deployed image classifier: it submits images and reads back the posterior probability vector over the victim’s K classes, paying a cost per query. It does not know the victim’s architecture or hyperparameters, the data used to train and evaluate the victim, or the semantics of the K output classes (which index corresponds to which label). It acts only at inference time and changes nothing about the victim’s training. The adversary supplies its own pool of images, drawn from public sources that may share little or nothing with the victim’s training distribution, and picks its own, possibly different, architecture for the knockoff. Its objective is a substitute that performs the victim’s task well, measured on the victim’s held-out test distribution, while keeping the number of paid queries small. The defender’s implicit claim, that keeping the model and its data confidential protects the model’s value, is what the attack tests; the paper also considers a defender that degrades its responses by truncating or rounding the probabilities it returns.

Basic Background

Image classification with convolutional networks

The victim and the knockoff are convolutional neural networks (CNNs) for image classification: each maps an image to a posterior probability distribution over a fixed set of K classes, and is trained by stochastic gradient descent on a cross-entropy loss against labels. The experiments use standard backbones, including ResNet (He et al., 2016) and VGG (Simonyan and Zisserman, 2014), which function as interchangeable architecture choices.

Transfer learning and ImageNet pretraining

Both the victim models and the knockoff are built by transfer learning: a backbone pretrained on the large ImageNet / ILSVRC dataset (Deng et al., 2009; Russakovsky et al., 2015) is then fine-tuned on a task. ImageNet plays a second role here as well, as one of the large public image pools from which an adversary can draw its queries.

Knowledge distillation

Knowledge distillation trains a student model to match a teacher’s output distribution rather than ground-truth labels alone (Hinton et al., 2015; Buciluǎ et al., 2006). Training a model on another model’s soft predictions over a set of inputs is equivalent, up to a constant, to minimizing the KL divergence between the two. Classic distillation assumes white-box access to the teacher and a shared task and training distribution.

Model extraction and black-box access

Model extraction attacks reproduce some asset of a deployed model from external access, typically the black-box query interface of a prediction API: inputs in, outputs out, with no view of weights or gradients. Different attacks target different assets: exact parameters, hyperparameters, the architecture, or the model’s functionality. A common distinction is fidelity (matching the victim prediction for prediction) versus accuracy (performing the underlying task well).

Query selection: active and reinforcement learning

Choosing which inputs to query under a budget is the subject of active learning, in particular pool-based active learning, where the learner picks queries from a fixed pool of unlabeled data (Settles and Craven, 2008). When the available feedback is an evaluative reward for chosen actions rather than a correct label, the natural framing is reinforcement learning; its simplest instance is the multi-armed bandit, which trades off exploring uncertain actions against exploiting ones already known to pay well (Sutton and Barto, 1998).

Reading guidance

• Section 3 (Problem Statement): what the adversary is denied, namely the victim’s internals, its train and test data, and the semantics of its output classes. Note that the last of these is an explicit modelling choice.
• Section 4.1.2 and Figure 4: the adaptive strategy’s reward, summing a margin-based certainty term, a diversity term, and a loss term. Note the justification given for each and how they are combined.
• Table 2: every multiplier is normalized against the victim black-box row, so a knockoff at 0.93x recovers that share of the victim’s accuracy. The knowledge distillation row is a second reference, the adversary querying the victim’s own training images; note where the knockoffs land against it.