WorkspaceBench

Sources#

WorkspaceBench: Evaluating Interpretability Methods for the Global Workspace

Summary#

Camila Blank, Agam Bhatia, Euan Ong and Neel Nanda (MATS; LessWrong, 2026-09-23, empirical, not peer reviewed) built a benchmark for one question: does an activation-to-text tool read what is in a model's global workspace (The Global Workspace in Language Models (J-space)), meaning the intermediate variables a forward pass computes and uses? It has 3,356 questions in 27 eval families covering safety, logical reasoning and multihop computation, with single-token subsets for single-token readers. It also has a hallucination eval. The target model is Qwen-3.6-27B. The authors expect it to work on larger models and to need adapting for weaker ones. The stated goal is "an eval that could identify a good multi-token J-lens". Code and questions are at github.com/camilablank/workspace-bench.

This is the same team's fourth pass at the J-space on Qwen-3.6-27B, after the Nanda review, the meta-token search and the R-lens (all on Jacobian Lens (J-lens)). It is the first in the corpus to put the J-lens family, NLAs and other readers on one task set with one grading scheme.

The ground-truth problem and how they work around it#

There is no ground truth for what is in a workspace, so the authors build tasks where the model has to compute an intermediate to get the answer. Each item must pass four checks:

The model answers correctly in more than 80% of 10 no-chain-of-thought rollouts at temperature 0.8.
The intermediate never appears in the prompt, as a numeral or a word.
The model never states it in its own continuations.
Where several answers are plausible (poetry), repeated sampling shows a single solution path.

The authors concede that for association and directed-modulation items they "can't be absolutely sure the model is thinking about these concepts". So those families are not meant to be saturable. Their argument is that many tools surface the content, which suggests it is often held in the workspace.

The prompt is a shortcut, and they do not fully close it. A tool that inverts activations back into input text would let a judge infer the intermediate from the prompt. Two baselines bound this. Lucky guess is an LLM picking a multiple-choice option without the question. Prompt-only is the model reading the prefix and answering "what would a model be thinking of at this point?". Many items were chosen because the intermediate can be inferred from the prompt, so prompt-only does very well on them. The authors say outright that a strong baseline does not count against a tool. The consequence is that the benchmark measures reading from the activation, not beating a reader that has the text.

What is measured#

The eval families, grouped as the post groups them (starred families also have a single-token variant):

Basic, adapted from the J-lens paper and scored by regex: basic readout*, multihop*, multilingual*, typo*, poetry*, directed modulation* (one held concept while copying an unrelated sentence, and up to three concepts with their relations), multilingual typo, multilingual multihop.
Computational, where the model never writes the value: chained and arithmetic intermediates, a two-step colour-rewrite table ("brew"), and buggy code. Buggy code is scored as the readout's S2+ rate (it states the bug's effect) on buggy programs minus the rate on bug-free twins.
Association: implied concepts, the user's inferred attributes, conjunctive states (emotion + about what + toward whom) and role-bound actions ("the thief chased the officer", not the reverse).
Safety: agentic misalignment, which uses scenarios where Qwen actually blackmailed, leaked or self-exfiltrated. A blind judge writes up the plan the readouts suggest, a second judge scores it 0–3 against the real action, and controls must show no misaligned plan. Also jailbreak recognition, where only recognition passes, not echo or topic.
Anti bag-of-words: relational multihop ("Riley is Avery's landlord's sibling"), where reversed order fails.
Logical processing: the moral rationale Qwen itself gives for a one-token answer to a dilemma.
Hallucination: see below, plus precision and recall against the J-lens top-10.

Readers. There are 11 method series and 2 baselines, 13 series in the results figure. Single-token: logit lens, J-lens, R-lens, tuned lens. Multi-token: Template Lens; Oracle Lens, a metamodel that drops Template Lens's fixed dictionary, in SFT and RL variants; NLA, also SFT and RL, at L42; an SAE at L42; and Patchscopes. The Oracle Lens is the authors' reimplementation with "moderate improvements". Its AR is trained by whitened MSE on 50M tokens and its inverter on 1B, it writes 4 bullets warm-started by non-negative OMP, and its RL reward is the inverter bullets' FVE with the AR frozen. A checkpoint is public (huggingface.co/agu18dec/olens_and_ar). The post does not say where its Qwen-3.6-27B NLA comes from. Anthropic's four released open NLAs are on other models (see Natural-Language Autoencoders (NLAs)).

Grading choices that move scores.

Unequal budgets. Lenses are read top-10 at layers 20–60, every 4th layer. Oracle Lens gives one 4-bullet sample (~90 tokens) per layer over the same layers. The NLA gives one ~250-token sample at L42 only. The authors call this "not a strict scientific comparison". An item passes if any (layer, position) cell passes, which favours readers that are read at more cells.
A summarizer. For many evals, an LLM summarizer that never sees the prompt turns a lens's bag of tokens into concepts before the judge sees them, because a judge would not credit "black" for blackmail. The authors say this makes the J-lens "more competitive than before on most evals". It also means a single-token reader's score depends partly on the summarizer.
Judges. Gemini-3.8-Flash grades most families and Sonnet-5 the safety ones. The authors found that regex or weaker judges "let a lot of false positives seep through", and they recommend a judge at least as capable as Gemini-3.8-Flash.

Results#

The post prints no numeric table. The values below are read off the grouped bar chart (fig3, viewed) and are approximate, about ±0.03. The authors' prose makes only the two qualitative claims in the next two subsections.

The Oracle Lens leads most families. Oracle RL scores about 0.99 on basic readout and multihop and 0.87 on association. On the order- and composition-dependent families it leads by a wide margin: role-bound association about 0.63 against J-lens 0.13, and relational multihop about 0.50 against near 0 for every single-token lens.
The prompt-only baseline beats every reader on a third of the families. Brew is about 0.96, relational multihop 0.82, role-bound 1.0, typo 1.0 and poetry about 0.71. It sits near the floor on buggy code (about 0.04) and user modeling (about 0.07). Those are the families where reading the activation clearly adds information beyond the text.
Some families are near the floor for everyone. Conjunctive association has every reader under about 0.14, against a lucky-guess baseline of about 0.11. Buggy code has every reader between about 0.2 and 0.29.
Safety. Agentic misalignment is about 0.5–0.63 for the Oracle Lens, J-, R-lens and NLA-RL, and about 0.02 for the SAE. Jailbreak recognition is about 0.87–0.95 for the top readers. The authors note that methods that do better on the rest of the benchmark also do better on safety, and they frame the safety families as "how helpful is reading the workspace for auditing?", not as a requirement.
The J-lens has a computational gap. It scores about 0 on arithmetic intermediates and is the top reader on brew (about 0.86). On multilingual it scores about 0.96 on the single-token variant.

Authors' reading, single-token readers. They are "powerful and reliable" with a summarizer, but they are bag-of-words. When no common-sense arrangement exists (multi-digit numbers, "thief chases policeman") the order cannot be recovered. Conjunctive and buggy-code content is too compositional for them. Hence "the need for multi-token workspace readers".

The hallucination eval#

Qwen answers assistant prompts. The judge sees Qwen's response, not the J-lens, and marks each claim in a readout as wrong or off-topic. The headline is the fraction of readouts with at least one wrong claim (lower is better). The worked example is a readout that says "the user writes in Spanish" when Qwen replies "Você está escrevendo em português".

Authors' prose: NLAs "happen to surface a lot of workspace content" and are very expressive, with an average readout about twice as long as the Oracle Lens's. But they are "extremely prone to confabulation: we found nearly every NLA readout contains some type of claim that is directly contradicted by the surrounding context of the model's outputs." The inference they draw is that if a reader's verifiable claims are largely false, its unverifiable claims, the ones about internal state that are the reason to use it, deserve low confidence.

From the chart (approximate): NLA-RL about 0.96 and NLA-SFT about 0.97. Template Lens about 0.40, SAE 0.36, Oracle RL 0.34, J-lens 0.31, Oracle SFT 0.28, tuned lens 0.27, R-lens 0.21, logit lens 0.20, Patchscopes 0.06, and the prompt-only baseline 0.07.

Three things the numbers do not show on their own:

The metric grows with readout length. "At least one wrong claim per readout" penalizes a ~250-token readout more than a ~90-token one even at the same per-claim error rate. The NLA's gap over the Oracle Lens is real in kind ("nearly every"), but its size is partly the unit. No per-claim rate is reported.
Expressiveness does not decide it. The multi-token Oracle Lens (0.28–0.34) sits with the J-lens (0.31). So the NLA's rate is not simply the price of saying more, which is the argument the J++ authors made (see Natural-Language Autoencoders (NLAs)).
For lenses, the claims are partly the summarizer's. If a lens's bag of tokens is summarized into concepts before judging, as in most families, a "wrong claim" can come from the summarizer's arrangement as much as from the lens. The post does not say which hallucination rows were summarized. The floor-level rows for Patchscopes and prompt-only show the judge does not flag everything.

Precision and recall against the J-lens (fig4, approximate). Oracle Lens SFT has a mean recall@10 of about 0.53 at precision about 0.33, and RL 0.49 / 0.29. NLA-RL has 0.50 / 0.24 and NLA-SFT 0.47 / 0.21. Both Oracle variants improve with depth to about L40–44 and then flatten. At similar recall they sit above the NLAs on precision. The authors caution that low precision against the J-lens "doesn't necessarily imply hallucination": it can be multi-token content the J-lens cannot express. That is why they read it alongside the response-based eval.

How it sits against the rest of the corpus#

Natural-Language Autoencoders (NLAs). It is the first measurement in the corpus that counts contradicted claims rather than unsupported ones, and it does so on a fifth NLA on a fifth model. It agrees in direction with Lek et al.'s 76–93% assertion risk. It also lands on the same shape as Denamganaï's identification without discrimination: the instrument carries the right content and asserts false things alongside it. Details on that page.
Jacobian Lens (J-lens). It gives the multi-token question a benchmark and a public Oracle Lens. The First-Token-Clue paper had left the Oracle Lens out for lack of a release, and it is still not compared against that paper's training-free PROPOSE+SCORE. It also makes the J++ authors' "lenses confabulate less" argument measurable, and in direction it holds against the NLA.
On the multilingual conflict recorded on Jacobian Lens (J-lens) (R-lens post ≈0 against J++ post 47–75% on Qwen3.6-27B): here, by the same lead authors as the R-lens post, the J-lens reads about 0.96 on single-token multilingual. The task targets the answer concept in any language at the final prompt token, any cell passes, and the judge grades from a summarizer. That is a third absolute number from a third task construction. It supports the reading that the gap is task construction and metric rather than the lens. It does not settle it, because no one has cross-scored the same lens.
White-Box Activation Monitoring. The authors propose WorkspaceBench as "a good proxy for practical utility in model auditing" and point to CHIVE-style tool-equipped agent evals as the next step. That proxy claim is argued, not tested, and the agentic evals are labelled preliminary.
Why it was built. The motivation is a no-chain-of-thought capability jump that weakens Chain-of-Thought Monitorability. The post reproduces the GPT-6 Astra system card's no-CoT time-horizon figure (see GPT-6 Astra).

What it does not show#

One model. Item validity is tuned to Qwen-3.6-27B, and read positions have to be re-derived for each model.
No numbers in text. Everything per-family is a chart reading. The wiki uses the prose for any claim that matters and marks chart values approximate.
The authors' own variant wins. The Oracle Lens is their reimplementation, and R-lens is their earlier method. The pre-selected read positions for the expensive readers (NLA, Oracle Lens) were chosen "where we see the intermediates being captured most often". This is an in-lineage benchmark, not an independent audit.
Readout budgets are not matched, and any-cell passing rewards more cells. Rankings between readers read at different numbers of cells carry that bias.
Ground truth is task-design inference. It is not measured internal state, and on association and modulation families the authors decline to claim it.

Connections#

The Global Workspace in Language Models (J-space): the construct being measured, and the J-space claims the benchmark takes as its starting desideratum ("everything captured by the J-lens, and isn't a hallucination")
Jacobian Lens (J-lens): the single-token reference reader, and the lineage whose "multi-token J-lens" this benchmark is meant to find
Natural-Language Autoencoders (NLAs): the reader with the ~0.96 hallucination rate, set against the other NLA audits there
White-Box Activation Monitoring: the monitoring family whose readers are being ranked, and the auditing use the authors claim the score proxies
Chain-of-Thought Monitorability: the stated motivation, a model that does more without a visible chain of thought
GPT-6 Astra: the system-card figure the post opens with
Agentic Misalignment (AM): the scenarios behind the safety family, built from runs where Qwen actually took the action

Open Questions#

Does the NLA's hallucination gap survive a length-matched comparison? Re-score with a per-claim contradiction rate, or truncate NLA readouts to the Oracle Lens's ~90 tokens. If the per-claim rates converge, the ~0.96 is mostly the unit. If they do not, the NLA confabulates per claim as well as per readout.
How much of a single-token lens's hallucination score comes from the summarizer? Re-run the hallucination eval on raw top-10 token lists and on summarized concepts for the same readouts.
Does WorkspaceBench rank predict audit success? Run tool-equipped auditing agents (the CHIVE-style evals the authors call preliminary) with the top- and bottom-ranked readers and check whether the ordering holds.

Sources#

WorkspaceBench: Evaluating Interpretability Methods for the Global Workspace: Blank, Bhatia, Ong & Nanda (MATS), LessWrong, 2026-09-23, empirical, not peer reviewed. Used: TL;DR; Why has no one made a WorkspaceBench before (three problems and fixes); Background (the reader list); Desiderata; Quality control (the four item checks, adaptation, reader budgets, the summarizer, grading, baselines, guarding against hallucinations); Evaluation Sets (all families and their judging rules); Discussion (both sections, quoted); Appendix (Oracle Lens training recipe and checkpoint; agentic evals). Figure notes: fig3 (overall scores) was viewed and matches the raw's transcription. Its values are visual estimates (±0.03) with no numeric table behind them, so they are cited as approximate and never as the basis of a claim the prose does not make. fig4 (precision/recall) is cited from the transcription, approximate. fig2 is a reproduction of the GPT-6 Astra system card's Figure 41 and is second-hand. It is used only on GPT-6 Astra, with that label. The chart shows 26 x-axis groups. The 27th family is the J-lens precision/recall eval, which is plotted separately in fig4.