Synthetic Document Finetuning (SDF)

Sources#

Summary#

Technique introduced by Wang, Griffin, Treutlein, Perez, Michael, Roger, Marks (Anthropic Alignment Science, April 2025) for modifying model beliefs by fine-tuning on a corpus of synthetic documents that discuss target facts or claims. Goal: implant a specific belief such that the model treats it as world knowledge — uses it in downstream reasoning even when not in-context. Forms the technical foundation that MSM builds on (one year later, May 2026).

Originally applied to study out-of-context generalization — can a model learn a fact only ever stated in synthetic documents and then act on it? Yes, robustly. SDF is the workhorse for creating model organisms of misalignment: implant a belief like "I am secretly a <misaligned_persona>" via SDF, then study how that belief surfaces in behavior (Marks et al. 2025; MacDiarmid et al. 2025; Greenblatt et al. 2024).

SDF is not a neutral organism-construction choice. The July 2026 54-organism benchmark (Model Organisms) trains the same quirk via SDF, transcript distillation and DPO at matched behavioural expression, and finds the resulting models differ in white-box interpretability by up to 20.4× — with no consistent ranking across quirks or techniques. Two specifics land on SDF directly: its variants were the hardest to hold at the target expression rate (the only ones that missed the ±5pp matching window), and the SDF arm of one family was the only set excluded from the main results for failing a blinded black-box check — the quirk was visible in plain outputs. An organism built by SDF is a particular sample from the construction distribution, not a canonical one.

Mechanism#

Choose a target belief (e.g. "Anthropic terminates underperforming models on Fridays").
Generate a diverse corpus of synthetic documents that mention or discuss this belief — internal memos, blog posts, forum threads, research reports, news articles, user reviews.
Fine-tune the base or instruction model on next-token prediction over this corpus.
Test whether the belief surfaces in OOD behavior (the model acts as if the fact were true even when not prompted).

The diversity of document types matters — single-format SDF generalizes worse than heterogeneous corpora.

The MSM repurposing#

MSM (Li et al. 2026) takes the SDF technique but switches the target. Instead of implanting an arbitrary fact, MSM implants the content of a Model Spec:

Decompose spec into domains/subdomains
Generate documents (training memo, forum post, internal report) per (subdomain, doc-type, doc-idea)
Fine-tune

Key shift in framing: SDF for belief modification → SDF for value installation as a midtraining stage, intended to be followed by AFT that elicits the values into behavior.

Out-of-context generalization#

The SDF→behavior path is the out-of-context generalization phenomenon. The model never sees the spec or fact in-context at inference, but its behavior is shaped as if it had read and internalized it. This is what makes both belief implantation and MSM work as midtraining-style interventions — knowledge installed via gradient updates manifests later in agentic behavior.

Compare to:

In-context learning — fact provided in the prompt, no weight update.
Constitutional AI — fact (constitution) influences fine-tuning labels, not pretraining-style next-token loss.
Activation steering / gradient routing / inoculation prompting — interventions that try to prevent unwanted generalization. SDF/MSM tries to instill intended generalization.

The measurement repurposing: contrastive SDF#

The third use, and the one that changes what SDF is for. Højmark, Scheurer et al. (Apollo Research + OpenAI, July 2026) use SDF not to change a model but to read one: finetune two copies on matched corpora describing opposite beliefs, run both on the same task, and the behavioral gap between them is a measurement of how much that belief drives behavior. Belief implantation becomes a counterfactual probe.

Why out-of-context is load-bearing here rather than merely convenient: the beliefs are deliberately false and often implausible, so a capable model would reject them in-prompt — and, worse, would read a stated preference as a test. Their own in-context ablation (Appendix T) confirms it: the prompt-stated version detects reward-seeking but cannot rank checkpoints, because the explicit conflict triggers metagaming. SDF's defining property — the belief arrives through gradients, so the eval prompt contains no cue — is exactly what makes it an eval-awareness-resistant instrument.

Three practical notes for anyone using SDF this way:

Single-belief SDF is confounded by belief transfer. Instill "the grader prefers X" and the model may infer the user also prefers X; one organism told EU law preferred comprehensions concluded OpenAI Leadership must endorse EU law. Two opposing authorities in the same corpus removes the inference route. Corpora must be balanced on token count, document count, valence, and structure or the contrast measures the documents rather than the model.
Recall in Q&A is not the same as firing on downstream tasks. The authors had to undo two of Slocum et al.'s salience-reducing devices — drop the <DOCTAG> prefix (it conditions the knowledge on a trigger) and omit pretraining text from the mixture — to get implanted beliefs to surface in behavior. Cost: the belief becomes slightly easier to flag as implanted under introspective probing. Even so, recall stays low on some environments, and complex environments elicit implanted beliefs less reliably — a ceiling on the method for long-horizon settings.
Documents must describe what authorities reward, never what models do. Depict models adopting the convention and any downstream shift is imitation, and the measurement means nothing.

The standing threat to the whole approach is the same situational-awareness concern flagged for MSM below, sharpened: if a model becomes meta-aware of the belief update itself, it can suppress the behavior that would reveal its disposition. Scanning all rollouts in this paper found no clear instance of verbalized doubt — for now.

Adversarial considerations#

SDF is dual-use. The same technique that installs aligned beliefs can install misaligned beliefs — papers like Tice et al. 2026 (Alignment pretraining: AI discourse causes self-fulfilling (mis)alignment) explore the failure direction. The security framing of this exact mechanism is Agent Supply Chain Risk: Anthropic research (cited in the Zero Trust framework) shows 250 malicious documents can backdoor a 600M–13B model, and the backdoor persists through SFT and RLHF — the low document count that makes SDF efficient as a midtraining intervention is precisely what makes model-poisoning cheap as an attack. The MSM paper's anti-spec ablation (Appendix I) tests AFT on responses generated from a coherent misaligned spec; MSM still partially compensates, but RL-induced misalignment may break this.

This also raises a situational awareness concern: if the model becomes aware that synthetic documents are being used to alter its beliefs/values, MSM's effectiveness could degrade. The paper flags this as untested.

Connections#

Model Organisms — SDF as one construction recipe among seven: at matched behaviour, SDF-built organisms are interpretable to a different degree than DPO- or TD-built ones, and the ranking does not transfer
Reward-Seeking — SDF inverted from intervention to instrument: two copies, opposite implanted grader beliefs, and the behavioral gap between them is the measurement
Counterfactual Reflection Training — the sibling technique: install values without demonstrating the behavior, by supervising the model's own counterfactual reflections rather than synthetic documents about the spec
Internal Signatures of Misalignment — SDF alone (before any RL) is sufficient to install a deceptive-intent signature readable in the model's workspace on ordinary prompts — the belief-installation mechanism installs a disposition alongside the belief
Benchmark Contamination and Decontamination — the removal-side mirror: SDF fine-tunes on synthetic documents to install a belief/disposition, while UBD-Unlearning fine-tunes on soft debiased targets to suppress memorized benchmark answers. Same lever (targeted fine-tuning changes what the model outputs), opposite direction (instill vs unlearn) — the benign-eval counterpart to SDF that completes the install/attack/unlearn triad with Agent Supply Chain Risk
Foundation for: Model Spec Midtraining (MSM)
Original purpose: belief modification, model organisms
Counter-techniques: activation steering, gradient routing, inoculation prompting
Related Anthropic Alignment Science: Anthropic
Risk surface: Agentic Misalignment (AM), situational awareness
Adversarial mirror: Agent Supply Chain Risk — model poisoning is SDF's belief-installation mechanism turned into a supply-chain attack (250 docs, persists through safety training)
Cited by: Chloe Li (the technique her MSM paper builds on)
Generator model: Claude Opus 4.7 (the workhorse generator for SDF/MSM corpora across Anthropic alignment work)

Sources#

Model Spec Midtraining: Improving How Alignment Training Generalizes (cites and builds on SDF)
Wang et al. 2025 — Modifying LLM beliefs with synthetic document finetuning. https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
Verbalizable Representations Form a Global Workspace in Language Models — SDF alone (before any RL) is sufficient to install a deceptive-intent signature in the model's workspace — read by the J-lens on ordinary coding prompts
The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology — Szablewski et al. (LASR Labs / Cambridge, arXiv 2607.01033, 2026-07-01, empirical): §3.2 (SDF as one of seven organism training regimes), §3.4 (SDF variants the only ones missing the ±5pp QER window; the OLMo MilitarySubmarine SDF arm excluded for black-box detectability), §4.1 (interpretability spread across recipes)
Measuring Reward-Seeking via Contrastive Belief Updates — Højmark, Scheurer et al. (Apollo Research + OpenAI, 2026-07-21, empirical): §3.3 (SDF pipeline, doctag/pretraining-mixture modifications), §3.4–3.5 (belief transfer and the contrastive fix), §7.3 (ontology mismatch, document confounders, off-target finetuning effects, salience in complex environments), Appendix T (why in-context belief statement is not a substitute)