Wiki Home
Gradient clipping
Definition
Gradient clipping rescales a gradient whose norm exceeds a threshold C down to norm C, leaving smaller gradients untouched. In ordinary deep learning it is a stabilizer against exploding gradients and is usually applied to the averaged batch gradient. In differentially private training the same operation does a different job and must be applied per example: clipping each example’s gradient before aggregation caps any single example’s influence on the update at C, giving the bounded sensitivity that the Gaussian mechanism requires. The threshold trades two errors: too small and the clipped average points away from the true gradient, too large and the noise (scaled to C) swamps the signal.
Papers that use this concept
- Deep Learning with Differential Privacy — per-example clipping is the sensitivity bound at the core of DP-SGD; the paper chooses C from the median of unclipped gradient norms.
- Unlocking the Power of Differentially Private Zeroth-order Optimization for Fine-tuning LLMs — clips an aggregated zeroth-order estimate rather than a first-order gradient, and ties the resulting clipping bias to the private optimizer’s convergence.