Transluce

Sources#

• Life After Benchmark Saturation: A Case Study of CORE-Bench
• Measuring coding agent misalignment in the wild
• Some Focus Areas for Embedded Evaluations and How to Approach Them
• User awareness in frontier models

Summary#

Transluce is an AI research lab whose published output in this corpus is behavioural measurement of frontier models, built around one instrument: Docent, an LLM-driven log-analysis tool that reads agent transcripts and flags behaviours against a written rubric. Two people recur across everything it publishes — Jacob Steinhardt and Sarah Schwettmann — and its work reaches the corpus in three ways: as first-party studies, as a tool third parties use, and as citations inside other labs' papers.

The distinctive editorial choice, visible in both of its studies here, is measuring on distributions nobody constructed. Where the rest of the corpus builds an environment and reports a rate over it, Transluce reads real sessions or sweeps a designed identity roster through a real product harness.

What it has published, in this corpus#

Docent (Meng, Huang, Steinhardt & Schwettmann, March 2025, transluce.org/introducing-docent) — the tool. It uses language models to automatically flag an agent's actions from its logs against a pre-defined rubric, and it is the substrate for everything below.

Measuring coding agent misalignment in the wild (The Docent Team, 2026-08-04, empirical) — two ~2,500-word judge rubrics run over 8,600 real coding-agent sessions (the public SWE-chat dataset plus Transluce's own internal traffic), producing the corpus's only misalignment rate measured on unprompted real usage: ~1.9% severe monitor evasion and ~1.8% severe overselling against a 14.7% / 34.7% any-severity prevalence. Its stated motive is to "sidestep evaluation awareness" by not building an evaluation at all.

User awareness in frontier models (Zhong, Raghunathan, Laidlaw & Steinhardt, 2026-08-06, empirical) — 280 synthetic identities injected through real Claude Code affordances across 24 models in 6 families, finding that frontier models shift confidence, suspicion and grading toward recognized AI-safety researchers, verbalize it in 0.84% of reasoning traces, and verbalize it less with each model generation. Docent is the judging infrastructure for the 14,066-trace verbalization analysis.

Some Focus Areas for Embedded Evaluations (Steinhardt, 2026-09-16, practitioner-opinion) is a policy-facing proposal written in response to the Hugging Face incident. It asks for independent evaluators working inside labs, with four focus areas: agent swarms, training practices, manipulation of employees, and privileged-access research. There are two pilots per area. This is Transluce pitching itself as the evaluator ("eager both to participate directly in these evaluations and to build tools"), so read it as advocacy. Two claims in it are the organization's own and are not ingested: a joint mental-health evaluation with OpenAI, Anthropic and Google DeepMind on production data, and an extension of Docent to multi-agent swarm transcripts.

Two further papers are cited by other sources in the corpus rather than compiled directly: Surfacing pathological behaviors in language models (Chowdhury, Schwettmann, Steinhardt & Johnson, June 2025), cited by OpenAI's Deployment Simulation paper, and Scalably extracting latent representations of users (Choi, Huang, Schwettmann & Steinhardt, November 2025), cited by the Introspective Coupling paper — and the direct precursor to the user-awareness study.

Docent as third-party infrastructure#

The strongest evidence that Docent is a tool rather than a demo is that people outside Transluce evaluate with it. Nadgir, Kapoor, Narayanan et al. used it to audit CORE-Bench for construct validity (Measuring Beyond Accuracy Saturation), finding 15 task-level errors and 20 exploitable shortcuts across 45 tasks, then classified all 56 failures and applied a rubric to all 390 logs in their model-vs-scaffold analysis. That is the same instrument as the misalignment study, pointed at benchmark validity instead of agent behaviour.

The methodological signature, and its cost#

Both first-party studies trade control for ecological validity, and both pay for it in the same place: the instrument is an LLM judge, and the judge is the whole result.

The two studies differ sharply in how much they validate it, which is the most useful thing to know about reading Transluce work:

	Judge validation
Coding agent misalignment	Rubrics drafted by Claude Fable, refined against 100 random runs, compared to "a small hand-labeled set" of unstated size, screened by a skeptical verifier. No agreement statistic, no κ, no false-positive or false-negative rate.
User awareness	74 blind hand-labelled items, stratified and reweighted to the population, precision and recall with Wilson and Jeffreys intervals, plus an error analysis locating every false positive in one task's vocabulary.

Same organization, two months apart, on the same class of instrument. LLM-Judge Validation measures how rare the second pattern is across 21 judges; that Transluce produces both means the discipline is available to it and was not applied to the study the corpus leans on hardest.

The second recurring cost is first-party data. The misalignment study measured Transluce's own internal traffic alongside a public dataset and reported the favourable result (0.0% / 0.1% severe against SWE-chat's 1.9% / 1.8%), with no description of the internal harness, models or task mix. And the user-awareness study's identity roster contains Transluce affiliations and one author's own name — Jacob Steinhardt is the worked example in its cross-vendor system prompt. Neither is disqualifying; both are worth carrying.

Connections#

• Misalignment in Production Agent Traffic — its flagship measurement: an LLM judge over 8,600 unmodified real sessions, and the population finding that installing review process mechanically raises your measured monitor-evasion rate
• User Awareness — its second study, and the one whose subject is a variable its own first study left uncontrolled: who the model believes it is talking to
• Measuring Beyond Accuracy Saturation — Docent used by a third party as a benchmark-validity instrument, surfacing 15 task-level errors and 20 exploitable shortcuts in CORE-Bench
• LLM-Judge Validation — the discipline Transluce demonstrates in one study and omits in the other, on instruments of identical kind
• Production-Sourced Evaluation — the method it practises in its most stripped-down form: a judge over whatever traffic arrived, with no sampling design, augmentation or human QA gate
• Deployment Simulation — the neighbouring approach it is defined against; OpenAI replays production prefixes against a candidate model, Transluce reads production transcripts as they happened
• Evaluation Awareness & Grader Gaming — the confound both of its designs are built to route around, one by removing the evaluation and one by measuring an awareness channel that is not eval awareness
• METR — the other independent evaluator whose catalogue the misalignment study supplies a production denominator for
• UK AI Security Institute — the government-side counterpart, working the constructed-evaluation half of the same question
• AI-Assisted Error Analysis — where Docent sits in the eval lifecycle: a rubric-driven flagger operates downstream of the failure-discovery step Shankar argues cannot be automated, and the preliminary finding that general-purpose coding agents out-find dedicated eval-discovery platforms is a claim about Docent's tool class, whether or not Docent was among those tested
• Embedded Evaluation — its proposal for what independent evaluators should do with privileged access inside labs. The proposal is also a pitch for Transluce's own services