Concepts

Background concepts the papers rely on. Each page is a short, self-contained explanation you can read on its own or reach from a paper that uses it.

Concept	Summary
activation-steering	Controlling an LLM at inference by adding, subtracting, or projecting out a concept direction in its activations; steering vectors and their limits.
adaptive-attack	An attack built against the specific defense it is evaluated on; the security-evaluation standard a defense claim rests on.
adversarial-examples	The evasion phenomenon: small input perturbations that flip a prediction.
adversarial-threat-model	How an adversary’s power is specified: perturbation set, knowledge, timing.
adversarial-training	Training on adversarially perturbed inputs as a defense.
automated-program-repair	Generating a patch that makes a faulty program pass a correctness oracle; generate-and-validate (GenProg), plausible-vs-correct, and learned neural repair.
backdoor-attacks	A trigger-fired hidden behavior trained into a model via data poisoning; the overcapacity that enables it and its reuse as the mechanism behind black-box model watermarking.
ciphertext-packing	Packing many plaintext values into the SIMD slots of one homomorphic ciphertext so a single add or multiply acts on all of them; slot rotations and the amortization and communication savings it buys.
code-language-models	Transformers pretrained on source code that complete or generate programs from a prompt; Codex, code completion, and byte-pair tokenization.
contrastive-prompt-pairs	Eliciting a concept’s direction from a matched pair of prompts that differ only in the target concept, by differencing their activations.
convolutional-neural-network	CNNs for image classification: learned convolutional filters, pooling, a softmax head, and the standard backbones (AlexNet, VGG, ResNet, DenseNet).
cross-entropy	The negative log probability a model assigns to the correct answer; the default training objective and the per-example privacy signal.
cryptographic-commitment	A binding, hiding token published for a value and opened later; with a time-stamped public log it gives proof of priority for ownership claims.
data-poisoning	Corrupting training data to change what a model learns: availability vs targeted/integrity poisoning, backdoors as the trigger-conditioned subclass, and why training-set sanitization defenses presume data access.
decoding-strategies	How an LLM turns next-token distributions into text: greedy, temperature, top-k, top-p, and prefilling.
dense-retrieval	Retrieving documents by embedding the query and passages and ranking by vector similarity (top-k); the neural alternative to BM25 and the retriever inside RAG.
differential-privacy	The formal bound on any single example’s influence; DP-SGD and the epsilon guarantee.
direct-preference-optimization	Aligning an LM to pairwise preferences directly, without a separate reward model or RL loop; the DPO alternative to RLHF.
empirical-risk-minimization	Average training loss as a proxy for expected loss; the baseline objective.
federated-learning	Training one shared model across clients that keep their data local, exchanging updates with a server; local vs central DP and why locality alone is not privacy.
fgsm	The one-step gradient-sign attack and its role as a baseline.
fine-tuning	Continuing training of a pretrained model on a smaller task- or behavior-specific dataset; full vs parameter-efficient (LoRA) updates, and how it undoes earlier stages.
functionality-stealing	The model-extraction goal of a substitute that performs the victim’s task, scored on task accuracy rather than on agreeing with the victim.
gaussian-mechanism	Sensitivity and noise calibrated to it; the additive-noise recipe behind DP mechanisms.
gradient-clipping	Norm-rescaling gradients; a training stabilizer that, per example, becomes a sensitivity bound.
gradient-masking	The defense failure mode where gradients become useless but worst-case loss stays high.
group-fairness	Constraining a model to behave equitably across groups defined by a sensitive attribute; demographic parity, equalized odds, equality of opportunity, and pre/in/post-processing enforcement.
hardware-virtualization	Hypervisors and the Arm virtualization extensions (EL2, two-stage address translation, the IOMMU) that give each VM an isolated view of memory and devices; the substrate beneath virtualization-based TEEs.
homomorphic-encryption	Computing directly on ciphertexts so the result decrypts to the function of the plaintexts; partially / leveled / fully HE, bootstrapping, and the CKKS scheme for approximate real arithmetic.
indirect-prompt-injection	Prompt injection whose instructions arrive through content the application retrieves or is handed, planted by a third party.
instruction-tuning	Fine-tuning a base LM to follow natural-language instructions; the instruction-following objective.
jailbreak	A prompt that elicits behavior a safety-trained LLM was trained to refuse; distinct from adversarial examples.
kl-divergence	Asymmetric measure of how far one distribution is from a reference; the per-token unit of how much alignment moved the base model.
knowledge-distillation	Training a student to match a teacher’s output distribution (soft labels); its equivalence to minimizing KL and its link to model extraction.
language-model-pretraining	Autoregressive next-token pretraining; the pretraining objective, distribution, and base model.
likelihood-ratio-test	Deciding between two hypotheses by their likelihood ratio; Neyman-Pearson optimality.
linear-probing	Training a linear classifier on intermediate activations to test whether a property is linearly decodable; supervised concept directions.
linear-representation-hypothesis	The hypothesis that networks encode high-level concepts as linear directions in activation space; concept directions and word-vector analogies.
llm-tool-use	An LLM emitting calls to external tools and acting on their outputs; chaining calls toward a goal yields an agent.
llm-watermarking	Embedding a hidden, human-imperceptible signal in a generative model’s output text for provenance: decoding-time logit/sampling biasing vs post-hoc editing, statistical detection, and the contrast with model-weight watermarking.
lp-norms	ℓ0 / ℓ2 / ℓ-infinity balls as perturbation budgets and proxies for perceptual similarity.
machine-unlearning	Removing chosen training data’s influence from a trained model without full retraining; exact vs approximate unlearning and the right-to-be-forgotten motivation.
mechanistic-interpretability	Bottom-up interpretability that reverse-engineers a network into circuits of neurons or features; the foil to top-down representation-level analysis.
membership-inference	Predicting whether a specific example was in a model’s training set; the standard privacy audit.
memorization	Per-example fitting, random-label capacity, and the long-tail argument; the signal privacy attacks exploit.
model-explanations	Post-hoc explanations released with a prediction: feature attribution, influence-based, and counterfactual/recourse; what they reveal and the attack surface they open.
model-extraction	Attacks that reproduce a deployed model’s parameters, hyperparameters, architecture, or functionality from query access; the fidelity-versus-accuracy axes and the MLaaS setting.
model-fingerprinting	Identifying a model by properties it already has, typically its decision boundary, rather than by a marker planted at training time.
model-partitioning	Splitting a DNN across a trusted execution environment and an untrusted accelerator (GPU): shielding a privacy-sensitive subset in the enclave while offloading the rest in the clear for speed; on-device TSDP and the security-versus-utility trade-off.
model-watermarking	Embedding an owner-verifiable marker in a trained model; white-box (in the weights) vs black-box (backdoor-based trigger set) verification, and the assumption that the owner controls training.
mutual-information	How much observing one variable reveals about another; the KL divergence between the joint distribution and the product of the marginals, its entropy decomposition, and its role as an information-leakage and rate-distortion measure.
perplexity	How well an LM predicts a sequence (exponentiated cross-entropy); the LM quality metric and the per-example membership signal in extraction attacks.
privacy-budget	The (epsilon, delta) budget, composition theorems, accountants, and amplification by subsampling.
projected-gradient-descent	Gradient step plus projection; the constrained-optimization primitive behind the PGD attack.
prompt-injection	Adversarial instructions in a prompt or ingested content that override the intended task; direct and indirect.
quantization	Mapping floating-point weights and activations to low-precision integers under a shared scale; the deployment lever and the precondition for expressing a model in a proof circuit.
red-teaming	Probing a model for policy-violating outputs to inform safety training and as an evaluation benchmark.
reinforcement-learning	Training an agent to maximize a reward from chosen actions; policy gradients, the multi-armed bandit, and the gradient-bandit update.
remote-attestation	A hardware root of trust signs a measurement (a quote) of the code and data that ran, checked by a remote verifier; property-based attestation adds reference values from a trusted authority, and ML property cards (model card, datasheet, inference card).
retrieval-augmented-generation	Conditioning generation on documents fetched at inference time; retrieved text shares the context window with the instructions.
rlhf	Aligning an LM to human preferences via a reward model and policy optimization with a KL penalty to the base model.
robust-optimization	Min-max / saddle-point optimization and Danskin’s theorem.
roc-curves	TPR/FPR trade-offs, AUC, balanced accuracy, and why security evaluates at low false-positive rates.
safety-training	Training a deployed LLM to refuse restricted behaviors; refusal, RLHF-for-harmlessness, Constitutional AI.
secure-inference	Running model inference on a client’s private input against a server’s private model, revealing only the result; HE-, MPC-, and TEE-based instantiations and the interaction / bandwidth cost axes.
secure-multiparty-computation	Jointly computing a function over private inputs while revealing only the output; two-party computation, garbled circuits and secret sharing, semi-honest vs malicious adversaries, and simulation-based security.
shadow-models	Adversary-trained imitations of the target model used to calibrate membership inference attacks.
side-channel	Learning a secret from timing, cache state, memory-access patterns, or speculative execution rather than from outputs; the class TEE guarantees usually exclude.
singular-value-decomposition	Factoring a matrix into singular vectors and values; the rank, the four fundamental subspaces, and the null space as the inputs a matrix maps to zero.
softmax	Logits and the map that turns them into a probability distribution; the attention mixing weights, the API leakage decision, and the expensive nonlinearity under encryption.
software-vulnerability	A security-relevant code defect an attacker can exploit; MITRE’s CWE catalog and Top 25, and the concrete / localize / confirm distinction.
stochastic-gradient-descent	Minibatch gradient descent; the default optimizer and the source of attack gradients.
taint-tracking	Dynamic information-flow tracking (DIFT): labelling data at a source and propagating the label to every derived value, then enforcing a policy on where it may flow; software vs hardware (tag-bit) implementations and the confidentiality / integrity dual uses.
temperature	The scalar dividing the logits before the softmax; the decoding randomness knob and the distillation softening knob.
transfer-learning	Reusing representations across tasks; ImageNet pretraining then fine-tuning, for higher accuracy with less task-specific data.
transferability	Adversarial examples crafted on one model often fool another; the basis of black-box transfer attacks.
transformer	Stacked self-attention and feed-forward blocks with residual connections; the residual stream, and the nonlinearities that dominate encrypted and proved inference.
trusted-execution-environment	Hardware-isolated enclaves that shield code and data from a malicious OS, with remote attestation; Intel SGX, Arm TrustZone, AMD SEV, Sanctum, and their side-channel and run-time-attack limits.
white-box-black-box	Adversary knowledge assumptions, from full weights to query-only access.
zero-knowledge-proof	A proof that a computation on a secret witness was done correctly while revealing nothing about it; prover and verifier, completeness / soundness / zero-knowledge, interactive vs non-interactive (Fiat-Shamir), succinct proofs / zk-SNARKs, and the arithmetic-circuit model.
zero-shot-prompting	Eliciting a task from a pretrained LM through the prompt alone, no examples and no fine-tuning; the contrast with few-shot and with fine-tuning.
zeroth-order-optimization	Gradient-free optimization that estimates a descent direction from forward-pass loss differences along random directions; the two-point/SPSA estimator and the MeZO line for memory-efficient LLM fine-tuning.