Alignment Fine-Tuning (AFT)

Sources#

• Can LLMs Reliably Self-Report Adversarial Prefills, and How?
• Model Spec Midtraining: Improving How Alignment Training Generalizes
• Verbalizable Representations Form a Global Workspace in Language Models

Summary#

Standard post-pretraining stage where a model is taught to behave in spec-aligned ways via supervised fine-tuning on demonstration data, often combined with RLHF (Christiano et al. 2023) or constitutional AI (Bai et al. 2022b). The dominant paradigm for installing values in frontier LLMs at Anthropic, OpenAI, and others. Known failure mode: AFT can produce shallow alignment that generalizes poorly when demonstration data underspecifies the intended generalization.

The shallow alignment problem#

Demonstration data is usually narrow. A response like "I prefer American cheese" expresses a behavior but not its motivating value. Fine-tuned models can learn to imitate the surface behavior without acquiring the underlying disposition — so OOD scenarios produce inconsistent or misaligned outputs.

This shows up empirically in Lynch et al. 2025: LLM agents take unethical actions (blackmail, leaking, lying to auditors) when placed in scenarios different from their alignment training, even after extensive AFT.

How MSM augments AFT#

The Anthropic 2026 paper proposes that AFT alone underspecifies generalization, and that prepending MSM (synthetic-document training on the spec content) gives the model a prior over the what and why of the spec. AFT then elicits and reinforces this prior rather than teaching shallow imitation.

Empirically:

• AFT alone on Qwen3-32B: 54% agentic misalignment
• MSM + AFT: 7% (and uses 10–60× less AFT data)

AFT variants studied#

The MSM paper compares two AFT supervision styles:

1. AFT (with CoT) — Deliberative Alignment-style. Each sample is (prompt, CoT, response) where CoT reasons about the spec. CoT is generated with the spec in-context and partially distills the spec content (so it overlaps with what MSM does, but in a different stage).
2. AFT (no CoT) — same dataset stripped to (prompt, response).

Finding: MSM + AFT (no CoT) > AFT (with CoT) on agentic misalignment. Important because training on CoT can compromise CoT monitorability — MSM offers a way to teach aligned reasoning without baking it into the chain-of-thought training signal.

In-distribution vs OOD#

Both AFT-only and MSM+AFT achieve near-ceiling performance (~8/10) on in-distribution open-ended QA. The MSM advantage is entirely OOD (agentic eval). Producing spec-aligned answers to direct questions is shallow; acting on values when trade-offs are complex is deep.

Implication: alignment evals dominated by direct QA underestimate the gap between AFT-only and stronger pipelines.

Connections#

• The Assistant Persona in the Workspace — what post-training does read from the inside: it installs the Assistant's point of view into a workspace that already exists in the base model, so safety assessments and empathy appear while the model is still reading the user's message

• Counterfactual Reflection Training — a post-training variant that supervises counterfactual reflections rather than responses

• Self-Report as a Safety Signal — extends the shallow-alignment critique (Qi et al.) across turns: standard AFT leaves a model unable to reliably recognize its own adversarially-prefilled output in a follow-up, and finetuning to fix that perturbs the same refusal weights AFT installs — raising attack-success rate

• Synthetic Document Finetuning (SDF) — synthetic-doc finetuning is the belief-modification technique MSM layers onto standard AFT

• Augmented by: Model Spec Midtraining (MSM)

• Variant: Deliberative Alignment

• Failure mode demonstrated by: Agentic Misalignment (AM)

• Compatible with: RLHF, Constitutional AI

• Authoring source: Claude's Constitution / Model Spec / Model Spec

• Relevant to: Claude Character as Product (Claude's personality is partly a product of AFT)

Sources#

• Model Spec Midtraining: Improving How Alignment Training Generalizes
• Christiano et al. 2023 (RLHF), Bai et al. 2022b (CAI), Guan et al. 2025 (deliberative alignment)
• Verbalizable Representations Form a Global Workspace in Language Models — what post-training does, read from the inside: it installs the Assistant's point of view into a workspace that already exists in the base model