Wiki Home

SoK: Unintended Interactions among Machine Learning Defenses and Risks

Paper: IEEE S&P 2024

High-level overview

A machine learning model faces many risks at once, and each risk has its own defenses. Deploying a defense against one risk routinely changes the model’s susceptibility to a different risk, sometimes lowering it and sometimes raising it. Adding adversarial training to resist evasion, for instance, makes membership inference easier (Song et al., 2019; Hayes, 2020). This systematization of knowledge (SoK) collects these unintended interactions and explains them within one framework.

The framework rests on a single conjecture: overfitting and memorization, two properties of how a model fits its training data, are the common causes underlying the interactions. A defense takes effect by inducing or suppressing overfitting or memorization, and that same change moves the model’s exposure to other risks. The framework names a small set of underlying factors that govern overfitting and memorization, drawn from the training data, the training algorithm, and the model. The paper surveys the published interactions against this framework, then distills a guideline for conjecturing the direction of an interaction not yet studied.

The framework accommodates the reported interactions, spanning seven defenses and eight risks. Using the guideline, the paper conjectures two previously unstudied interactions and validates them empirically: enforcing group fairness lowers the success of data reconstruction, and releasing model explanations raises the success of distribution (property) inference.

Threat Model: A SoK has no single threat model; its scope is a population of settings. The defenses are deployed by a model owner: adversarial training, outlier removal, differential privacy, group fairness, model watermarking, fingerprinting, and the release of explanations. The risks are pursued by an adversary or arise as a property of the model: evasion, poisoning, unauthorized model ownership through model extraction, membership inference, data reconstruction, attribute inference, distribution inference, and discriminatory behaviour. The settings range over white-box and black-box access, train-time and inference-time action, and take classifiers as the primary object with generative models discussed as an extension. The framework’s claim is explanatory and predictive rather than adversarial: overfitting, memorization, and the factors that drive them determine whether deploying a given defense increases or decreases susceptibility to a given risk. The framework asserts correlation, not causation, and its conjectures are checked empirically.

Basic Background

Risks and defenses in trustworthy machine learning

Trustworthy machine learning studies a model’s exposure across security, privacy, and fairness. Security risks include evasion, where crafted perturbations force a misclassification (Goodfellow et al., 2015); data poisoning and backdoors, which corrupt training data to control predictions; and unauthorized model ownership through model extraction, which rebuilds a deployed model’s functionality from query access (Knockoff Nets; Orekondy et al., 2019). Privacy risks include membership inference, which tests whether a record was in the training set (likelihood-ratio formulation; Shokri et al., 2017; Carlini et al., 2022), along with data reconstruction (model inversion), attribute inference, and distribution (property) inference; for generative models the privacy failure is verbatim reproduction of training data (Carlini et al., 2021). Fairness concerns a model’s discriminatory behaviour across sensitive subgroups. The corresponding defenses include adversarial training (PGD; Madry et al., 2018), outlier removal, differential privacy via DP-SGD (DP-SGD; Abadi et al., 2016), model watermarking and fingerprinting for ownership (DAWN; Szyller et al., 2019), group fairness constraints, and model explanations.

Overfitting and memorization

A model overfits when it fits its training set more closely than the underlying distribution; the gap between training and test loss, the generalization error, measures it. Memorization is the per-example version: a model can fit individual records, especially outliers and the long tail of the data distribution, and some such memorization is necessary for accuracy (Feldman, 2020; Zhang et al., 2017). Both leave traces in what an adversary can observe, namely the model’s predictions, its intermediate activations, its gradients, and its parameters. Most of the risks above feed on those traces.

Group fairness and explanations

Group fairness constrains a model to behave equitably across groups defined by a sensitive attribute, under statistical metrics such as demographic parity or equalized odds (Hardt et al., 2016), enforced by pre-processing the data, constraining the objective, or post-processing the outputs. Model explanations release auxiliary information with each prediction: a feature attribution, the influential training records behind a prediction, or a counterfactual that would change the outcome. Neither defense requires retraining, and both expose information about the model’s decision boundary and training data.

Reading guidance

• Section 3, the conjecture stated at its head and unpacked in 3.1 to 3.3: overfitting and memorization are taken as the two underlying causes of every interaction in the survey. The justification for picking those two is stated briefly; note what it rests on, since the whole framework inherits it.
• Section 4.4, guidelines G1 to G4: how the direction of an interaction is inferred from a shared factor, and how a dominant factor is chosen when two factors disagree.
• Table 3 with its legend: each underlying factor in a cell is marked empirically shown, theoretically shown, or only conjectured, and exceptions are listed separately. Note how many carry the conjecture mark.