Wiki Home

Activation steering

Definition

Activation steering controls a model’s behavior by intervening on its hidden activations at inference, rather than by changing the prompt or the weights. A concept direction is added to or subtracted from the activations at one or more layers to strengthen or suppress that concept in the output, or projected out to remove it. Difference vectors between activations on paired prompts give such directions without any optimization (Turner et al., 2023), and steering a truthfulness direction has been used to make a model’s answers more truthful (Li et al., 2023). The intervention is cheap and targeted, which makes it attractive for safety; a single direction need not transfer across contexts, and a steer can perturb unrelated behavior.

Papers that use this concept

No reading-companion page currently uses this concept.

See also

• Linear representation hypothesis
• Contrastive prompt pairs
• Red teaming (language models)

References

• Li, K., Patel, O., Viégas, F., Pfister, H., and Wattenberg, M. “Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.” 2023.
• Turner, A., Thiergart, L., Udell, D., Leech, G., Mini, U., and MacDiarmid, M. “Activation Addition: Steering Language Models Without Optimization.” arXiv:2308.10248, 2023.