Wiki Home

Towards Deep Learning Models Resistant to Adversarial Attacks

Paper: ICLR 2018

High-level overview

Deep image classifiers can be fooled by adversarial examples: inputs perturbed by amounts too small for a human to notice but large enough to flip the model’s prediction. This paper reframes attacks and defenses as the two halves of one min-max (saddle-point) problem from robust optimization. An inner maximization finds the worst-case perturbation of an input within an allowed set, which is what an attack is. An outer minimization trains the network against that worst case, which is what adversarial training is. Several of the paper’s terms are worth decoding before they appear. “Adversary” here usually names an algorithm rather than a person. “First-order” says which derivatives the attack is allowed: only the first, the gradient of the loss with respect to the input pixels, which says which way to move to raise the loss and nothing about how the surface curves. “Saddle point” names the kind of solution a min-max problem asks for, a point that is a minimum in one variable (the weights) and a maximum in the other (the perturbation) at the same time. “Robust accuracy” is accuracy measured after the attacker has perturbed every test input, so it sits below clean accuracy on the same model.

The paper reports that the inner maximization is tractable in practice despite being non-concave, meaning its loss surface over the perturbation has many separate local maxima with no guarantee that any method finds the highest. Projected gradient descent (PGD), rerun from many random starting points inside the allowed set (“random restarts”), behaves as the strongest first-order attack. It reports that robust classification needs noticeably larger networks than clean classification. Training against the PGD adversary yields models that withstand the strongest attacks the authors evaluate, above 89% robust accuracy on MNIST and around 46% on CIFAR-10, and the trained models were released as public attack challenges. The method is now called PGD adversarial training or Madry-style adversarial training, it remains the reference baseline for empirical defenses, and “PGD” without qualification usually means this paper’s attack.

Threat Model: An evasion (test-time) setting. The adversary perturbs individual inputs at classification time; the training data and the model are untouched. What bounds the adversary is a set of allowed perturbations around each input, and the requirement on that set is perceptual: a perturbed input must still look like the original to a human, so that its true label is unchanged. The paper instantiates that set as the ℓ-infinity ball of radius epsilon, so no pixel may change by more than epsilon (0.3 on MNIST in [0,1] scale, 8 on CIFAR-10 in [0,255] scale), and treats this as one workable formalization of perceptual similarity while leaving richer ones open. The headline setting is white-box: the adversary knows the architecture and the weights and can compute gradients. Weaker black-box (transfer) adversaries are evaluated as well. The defender’s claim is accuracy under the worst allowed perturbation, argued for the class of first-order adversaries.

Basic Background

Training a classifier

A neural network classifier maps an input in R^d to one of k labels and is trained by empirical risk minimization: minimizing the average cross-entropy loss over the training set, as a proxy for the expected loss over the data distribution. The minimization runs by stochastic gradient descent, which estimates the gradient of the loss with respect to the parameters on a small batch and steps against it. The same backpropagation machinery also yields the gradient of the loss with respect to the input, and that input gradient is what gradient-based attacks use.

Measuring small perturbations

Perturbation size is measured with ℓp norms. The ℓ-infinity norm of a perturbation is its largest per-coordinate change, so the ℓ-infinity ball of radius epsilon contains every image whose pixels each differ by at most epsilon from the original. The ball is a stand-in for perceptual similarity, which has no closed form.

The adversarial setting

An adversarial example is a correctly classified input perturbed, within the allowed ball, so that the model misclassifies it (Szegedy et al., 2014; Goodfellow et al., 2015). A threat model makes “robust” falsifiable by fixing the perturbation set and the adversary’s knowledge: white-box adversaries see weights and gradients, black-box adversaries only query or transfer. Transferability, the fact that adversarial examples crafted on one model often fool another, is what makes black-box attacks practical.

Attack and defense primitives

The Fast Gradient Sign Method builds an ℓ-infinity adversarial example in one gradient step (Goodfellow et al., 2015). Taking several smaller steps and projecting back into the ball after each one is projected gradient descent, the standard method for constrained optimization, and it finds higher-loss points than a single step (Kurakin et al., 2017). On the defense side, adversarial training trains on attacked inputs instead of clean ones (Goodfellow et al., 2015). A defense evaluated only against weak attacks can fail by gradient masking: the gradients become useless to the attacker without the worst-case loss actually being small.

Worst-case optimization

Robust optimization minimizes the worst-case loss over an uncertainty set, written as a min-max problem, and predates deep learning by decades (Wald, 1945; Ben-Tal et al., 2009). Danskin’s theorem connects the two halves: the gradient of the loss evaluated at an inner maximizer is a descent direction for the outer problem. Its assumptions, a smooth function and an exact inner maximum, do not hold for ReLU networks attacked approximately, so applying it there needs empirical justification.

Reading guidance

• Section 2, the paragraph fixing the perturbation set S, and Equation 2.1: where the threat model is pinned down. What S has to capture is perceptual similarity; the ℓ-infinity ball is what the paper puts in its place. That substitution is made in one sentence, on one citation, with richer notions of perceptual similarity deferred to future work. Note what is offered for it.
• Section 3.2: the claim that robustness against PGD implies robustness against every first-order adversary. It is stated as a conjecture supported by the concentration of PGD’s local maxima, and the guarantee rests on it.
• Table 2: the CIFAR10 robustness numbers. Each row is a different attack and source network, so white-box and transfer results sit in one column and are told apart only by the row label.