Wiki Home

Unlocking the Power of Differentially Private Zeroth-order Optimization for Fine-tuning LLMs

Paper: USENIX Security 2025

High-level overview

Fine-tuning a pretrained large language model on a sensitive dataset can leak that dataset: the released model is open to membership inference and training-data extraction. Differentially private fine-tuning removes that risk by adapting the model under an (epsilon, delta) guarantee that bounds any single fine-tuning record’s influence on what is released. The standard method, DP-SGD, works on per-example first-order gradients (Abadi et al., 2016), but computing those gradients by backpropagation through an LLM is memory-heavy, and for billion-parameter models the gradient step alone can exceed a high-end GPU. This paper fine-tunes under differential privacy with zeroth-order optimization instead: a gradient-free optimizer that estimates a descent direction from forward-pass loss differences along random perturbation directions, so it needs roughly the memory of inference rather than of backpropagation. The method, DP-AggZO, aggregates multiple per-record zeroth-order direction estimates before clipping and adding Gaussian noise, which concentrates the per-record update so DP’s required clipping injects far less bias than clipping a single high-variance estimate.

DP-AggZO improves the privacy-utility trade-off of private zeroth-order fine-tuning. Across a masked language model (RoBERTa) and autoregressive models (the OPT family), on classification and generation tasks, it consistently beats the prior differentially private zeroth-order baseline (DPZO) at the same privacy budget, and it matches or surpasses the state-of-the-art first-order private optimizer (DP-AdamW) while using far less memory; on the 355M-parameter model it leads DP-AdamW across the benchmark tasks. On the largest model studied, the first-order baselines run out of memory on a 96 GB GPU, while DP-AggZO fine-tunes within that budget, at the cost of extra forward passes per update and with memory held at the single-estimate DPZO level.

Threat Model: A defense under differential privacy, so the guarantee is attack-agnostic rather than tied to a named adversary strategy. The protected secret is the presence and content of any single record in the private fine-tuning dataset; adjacent datasets differ in one record (record-level DP). The adversary is as strong as the definition allows: it observes the released fine-tuned model, may query its outputs, knows the training algorithm and the random perturbation directions as public information, and may control every other fine-tuning record. The defender claims (epsilon, delta)-differential privacy for what the fine-tuning run releases, which bounds the success of any membership inference or data-reconstruction attempt against the fine-tuning data, present or future. As in DP-SGD, the privacy reasoning is on the noisy per-iteration update, not on the final parameters; the pretrained base model is treated as public.

Basic Background

Differential privacy and its machinery

Differential privacy (DP) is a property of a randomized algorithm: adding or removing any single input record changes the output distribution by at most a bounded factor, set by the privacy parameters (epsilon, delta) (Dwork et al., 2006). The standard way to make a function private is to add noise scaled to its sensitivity, the most one record can change the output; the Gaussian mechanism does this with normal noise. A quantity with no a priori sensitivity bound, such as a gradient or a loss-difference estimate, is first forced into one by clipping it to a fixed norm per record. Each private step spends part of a privacy budget; composition theorems and a privacy accountant bound the total leakage across the thousands of steps a fine-tuning run takes, and sampling each batch at random amplifies the per-step guarantee. Iterative training is commonly analyzed with Rényi differential privacy, an accounting framework that tracks the Rényi divergence between the output distributions on adjacent datasets and composes tightly (Mironov, 2017).

Training and fine-tuning a pretrained language model

A language model is first pretrained on a large corpus by next-token prediction, producing a base model. Adapting that base model to a specific task on a smaller labeled dataset is fine-tuning, the language-model instance of transfer learning. Fine-tuning minimizes the task loss with stochastic gradient descent or an adaptive variant such as AdamW, estimating the gradient on a minibatch and stepping against it. For neural networks the gradient comes from backpropagation, which caches intermediate activations and gradients and so costs several times the memory of a forward pass; for a large model this is the dominant memory expense of fine-tuning. The models in this paper, the masked language model RoBERTa and the autoregressive OPT family, are standard open-source pretrained checkpoints.

Zeroth-order optimization

Zeroth-order optimization replaces the exact gradient with an estimate built only from loss values. For a random direction z, the scaled loss difference along z approximates the directional derivative, and multiplying it by z gives an estimate of the gradient that is correct in expectation as the perturbation scale shrinks. This two-point, simultaneous-perturbation estimator dates to stochastic approximation (Spall, 1992). Applied to LLM fine-tuning as MeZO, it needs only forward passes and stores a random seed rather than the perturbation vector, cutting memory to roughly that of inference, up to an order of magnitude below backpropagation (Malladi et al., 2023). The estimate is high-variance, since a few random directions probe a parameter space of millions of dimensions, so zeroth-order methods converge more slowly than first-order ones.

What private fine-tuning defends against

A fine-tuned model can leak its fine-tuning data. Membership inference predicts whether a specific record was in the training set (Shokri et al., 2017), and derived as a likelihood-ratio test it identifies members reliably even at low false-positive rates (LiRA; Carlini et al., 2022). Large language models also memorize individual training sequences verbatim, which a black-box attacker can pull back out (training-data extraction; Carlini et al., 2021). Differential privacy is the principled counter to both: bounding any record’s influence provably caps the success of any such attack, which is the guarantee this line of work targets.

Reading guidance

• Section 5.1, the input list of Algorithm 1: what the privacy guarantee is taken over, including the Laplace noise spent on releasing the dataset size n itself. Note what the adversary is assumed to observe.
• Section 4.1, Figure 1, with Lemma 3 in Section 5.2: the distribution of zeroth-order loss-difference estimates against the clipping thresholds prior work recommended, and the norm bound on the aggregated direction that in turn bounds the clipping error. Note how heavy that tail is next to those thresholds.
• Table 2, with Table 3 for peak memory on RoBERTa: utility on OPT-1.3B and OPT-6.7B against DP-AdamW and DPZO. The OOM cells are the claim; note which baselines fail to run at all rather than run worse.