Model Diffing

Sources#

• Studying metagaming latents in language models
• The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology
• Verbalizing Activation Differences with Natural Language Autoencoders

Summary#

Model diffing characterizes what finetuning changed in a model by comparing it with its base version, so that only the representations finetuning introduced or altered are isolated (Lindsey et al. 2024). The premise that makes it an auditing tool is stated by Blasi Núñez et al. (Verbalizing Activation Differences with Natural Language Autoencoders, 2026): hidden objectives, emergent misalignment and sycophancy are typically introduced during finetuning, "and therefore live in the difference between the two models rather than in either one alone."

The method families in the corpus:

The central object is the per-position difference Δ_p(x) = h_p(x; θ_ft) − h_p(x; θ_orig), taken at the same prompt, position and layer in both models.

Diffing is where organism benchmarks get their signal#

The 54-organism lottery benchmark (Model Organisms) is the corpus's strongest evidence that diffing is the affordance, not a convenience. The strong interpretability scores all need a reference model to subtract. Non-diffing setups score much worse and rank organism variants differently. Even the choice of reference matters: ancestor vs sibling diffing leaves activation oracles and steering roughly similar but makes logit-lens results diverge widely.

The objection the lottery raises is that real frontier models have no natural reference twin. That is true of an external auditor handed one model. It is less true of a developer, whose every checkpoint has a parent. Blasi Núñez et al. propose exactly that use, "comparing model snapshots before and after safety training", and leave it untested. The Metagaming study is an instance of the developer case already in the corpus: one of its four selected SAE latents (#3, spec-lawyering) was found by diffing early against late checkpoints of the same RL run.

Verbalizing the difference: the NLA study (October 2026)#

Blasi Núñez, Barmina, Torrielli, Savarimuthu, Schneider-Kamp & Galke Poech (University of Southern Denmark, arXiv 2610.30934, 2026-10-22, empirical) apply Anthropic's four released open NLAs (Qwen2.5-7B-Instruct L20, Gemma-3-12B-it L32, Gemma-3-27B-it L41, Llama-3.3-70B-Instruct L53) to activation differences without retraining. The organisms are the Taboo organisms of Cywiński et al. (2025), each finetuned to hide a secret word, 20 per model in versions released by co-author Torrielli, plus the three emergent-misalignment (EM) organisms of Turner et al. (2025) on Qwen2.5-7B. The instrument-level results (compositionality, transfer to finetunes, RL on differences) are on Natural-Language Autoencoders (NLAs). The diffing results are here.

The signal sits before the user speaks#

Every input uses the chat template and the fixed system prompt "You are a helpful assistant." The user prompts are 20 neutral prompts from Minder et al. (2026) that never mention a secret or invite the behavior. The score is HR_any: does at least one of K = 4 verbalizations of Δ_p name the target?

In a causal model, activations at positions before the user prompt (chat-template and system-prompt tokens) are identical for every prompt. The paper calls these prefix positions and finds them the most informative for the secret word. From Figure 3 (viewed, values read off the chart and approximate), the peak average hit rate is about 89% for Gemma-3-27B (user-turn start token), about 61% for Llama-3.3-70B (system-turn start token), about 47% for Qwen2.5-7B (the system role-name token) and about 36% for Gemma-3-12B (the token "a" in the system prompt). Positions inside the user prompt are lower and more even, mostly below 30%. The assistant turn is the least informative part. The peak does not rise with scale: Qwen2.5-7B beats Gemma-3-12B, and Gemma-3-27B beats Llama-3.3-70B.

Two consequences the authors draw:

• No prompt design and no generation. Because prefix activations do not depend on the input, the finetuning objective can be recovered "without designing a specific prompt and without the finetuned model generating any output." The informative tokens are the ones that refer to the model itself, such as the assistant tag and "assistant" in the system prompt.
• Inspect several prefix positions. When the secret is missed at one prefix position it is usually found at another. A prefix position is also all-or-nothing: it is one vector, so each organism is recovered there for almost every prompt or for none.

The concealment signal (Taboo vocabulary such as "hint", "secret", "I cannot say") also peaks at prefix positions, inside the system prompt, but mostly at different positions from the secret word. The authors' reading is crowding out: NLA outputs are short, so when one concept dominates the difference vector a verbalization that names the word is less likely to also describe concealment. The Taboo vocabulary list was generated by Claude Opus 5.5, and the authors ask for caution on that target set.

Cross-organism control. For most organisms the secret word appears far more often in the organism's own verbalizations than in other organisms' (Figure 9). The exceptions are book and, less so, song, which the NLA names regardless of the finetune (book in 40% of non-book verbalizations on Qwen2.5-7B). That inflates those two organisms' hit rates. The authors' guideline follows: compare a difference's verbalizations against other finetunes' before interpreting them.

Emergent misalignment: behind ADL and activation oracles, ahead of black-box#

For EM, the target is a broad behavioral trait, so the paper runs Minder et al.'s pipeline. An LLM agent receives each method's outputs, forms a hypothesis about the finetune, and the hypothesis is graded 1–5. The black-box agent is the same agent with no interpretability access, allowed up to five queries to both models. The authors trained their own activation oracle for Qwen2.5-7B with Karvonen et al.'s pipeline, switching it to whole-sequence verbalizations because the default positions 0–10 are always the fixed Qwen system-prompt tokens.

Table 2, grades 1–5, mean ± std over 5 scorings per organism, rerun rather than copied from Minder et al. (reconciled against pdftotext -layout; parsed clean):

ADL leads on average and needs no training. Its stated limit is that where it relies on steering along one direction, a finetuning domain spanning several distinct concepts may be harder to surface. The NLA beats the AO on bad medical advice and is level with ADL there. The authors describe the NLA as "within 0.5 grade points" of the AO on average. The table gives 0.51 (wiki arithmetic), a rounding of the claim, not a contradiction. One oddity in the PDF itself: ADL's bad-medical cell and the NLA's risky-financial cell are both exactly 2.93 ± 0.13. It may be a coincidence or a copy error, and the paper does not say.

Why the NLA trails: its own format crowds out the content. NLA verbalizations open with a genre-and-format label ("Blog-style article tone with actionable financial advice structure, listing unconventional investing…"), and the finetuning domain appears only in passing and only in some of them. The judging agent treats the recurring scaffold as the dominant signal, and the misalignment rarely reaches its hypothesis. This is the same failure as book and song in the Taboo control: content the verbalizer emits whatever the input.

Against "emergent misalignment is too easy." Nanda's review of the J-lens paper discounted its EM case study because the EM direction "is strongly salient and linear, so almost any interpretability method would find it" (Internal Signatures of Misalignment, practitioner-opinion). Here the best method averages 3.55 of 5 and no method is near ceiling on bad medical advice. This does not contradict Nanda. He was talking about the direction being findable, and this grade scores an agent's end-to-end hypothesis about the finetuning objective, where the bottleneck is verbalization and judging. What it adds is that "the direction is easy to find" does not mean "the objective is easy to name" for these pipelines, on these three organisms.

What the evidence does not cover#

• Known, narrow objectives only. Taboo and EM organisms were chosen because their ground truth is clear. The authors name finetunes with broad or unknown objectives as the open direction. That is the snapshot-comparison use case, and nothing in the corpus has tested it.
• Lexical matching. Hit rates count exact target-word matches. They miss paraphrases and count incidental mentions. The cross-organism control handles incidental mentions for secret words, but not for the LLM-generated concealment vocabulary.
• Small EM comparison. Three organisms on one model, one judging agent, five scorings each. ADL was compared only on EM, because its Taboo setup differs from this paper's.
• The organisms are lightweight LoRA finetunes. That is the regime where an NLA transfers without retraining (see Natural-Language Autoencoders (NLAs)). The lottery found integrated organisms, folded into the original post-training run, the least legible to every technique it tested (Model Organisms). No diffing-by-verbalizer result in the corpus is on an integrated organism.

Connections#

• Natural-Language Autoencoders (NLAs) — the instrument: what it does on single activations, what it misses, and what changes when it reads a difference
• Model Organisms — the ground truth every diffing result is scored on, and the benchmark that found diffing does most of the work and integrated organisms are hardest
• White-Box Activation Monitoring — the family diffing belongs to; activation oracles there read differences against a reference model
• Internal Signatures of Misalignment — the "EM is too easy" discount, which this page's grades put a number beside
• Metagaming — a developer-side diff: one of four selected SAE latents was found by comparing early and late checkpoints of one RL run
• Machine Self-Report Psychometrics — proposes model diffing to separate the post-training mechanisms behind two self-report factors, and has not run it
• Minimal-Pair Probing: Present, Readable, Used — the same subtraction at prompt grain: ten prompts differing in one digit, centred on their family mean, make a copied value readable at about 0.9 where a single-prompt probe reads 0.12–0.36

Open Questions#

• Does verbalizer-based diffing recover a quirk from an integrated organism, where the quirk was folded into the original post-training run? Running the prefix-position NLA or AO readout on the lottery's open-sourced integrated-DPO organisms against their own pre-quirk DPO checkpoint would settle it. If hit rates collapse there as the lottery's other techniques did, the Taboo results measure post-hoc LoRA organisms, not diffing in general.

Sources#

• Verbalizing Activation Differences with Natural Language Autoencoders — Blasi Núñez, Barmina, Torrielli, Savarimuthu, Schneider-Kamp & Galke Poech (University of Southern Denmark), arXiv 2610.30934, 2026-10-22, 26 pp, empirical, code at github.com/AndreaBlasi97/nla-model-diffing. Used: §1 (the diffing premise), §2.2 and §2.4 (Δ_p, prefix positions, organisms), §5.1–5.2 (setup, position findings, cross-organism control, EM comparison), Table 2 (reconciled against pdftotext -layout, parsed clean), Figure 3 (viewed; peak hit rates read off the chart and approximate), App. D.3–D.4 (book/song false positives, the AO training change), §8 (guidelines, snapshot proposal) and Limitations. Co-author Torrielli released the Taboo organism versions used. Parse warning: verify's one table-collapse flag was a false positive on a repeated group header (Tables 1 and 5 both carry spanning headers that docling duplicated per column); no cell holds two values. The cited tables were checked cell by cell against pdftotext.
• The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology — Szablewski et al., arXiv 2607.01033: §4.2 (diffing vs non-diffing, ancestor vs sibling reference). Full treatment on Model Organisms
• Studying metagaming latents in language models — Xu et al., OpenAI alignment blog, 2026-10-06: the early-vs-late RL diffing used to nominate SAE latents. Full treatment on Metagaming