H
Howardism
Howardism · Vol. 03Plate II · No. 02

Interpretability, in order.

Notes12DomainInterpretabilityOpen Qs33Newest23 Aug 2026Oldest7 Jun 2026

Reading model internals: activations and the global workspace.

Map of Content for the interpretability domain — 12 concepts. Reading model internals: the global workspace, the Jacobian lens, activation monitoring, internal signatures of misalignment, and the interference weights left behind by superposition. Curated entry point; see Home for all domains.

  • Access-Consciousness Indicators in AI — The consciousness question the workspace paper deliberately does and doesn't answer: it tests functional indicator properties (global workspace, higher-order, attention schema, recurrent processing) against a concrete inspectable structure, takes no position on phenomenal experience — and finds that ablating the J-space flattens the model's experiential reports while leaving its coherence intact
  • The Assistant Persona in the Workspace — Post-training installs the Assistant's point of view into a workspace that already exists in the base model: safety assessments and empathy appear while the model is still reading the user's message, and it internally flags its own outputs — disclaimer/fictional when roleplaying, an all-caps BUT when prefilled against its own preferences, damn when it fails to suppress a thought
  • Automatic vs. Flexible Cognition in LLMs — The selectivity result: a model can parse, classify, continue text and detect anomalies with its workspace suppressed, but loses multi-hop reasoning, translation, analogy and summarization — and chain-of-thought partially rescues it, because writing the intermediate down externalizes what the model would otherwise have to hold internally
  • Counterfactual Reflection Training — Train the model to write constitution-grounded reflections if interrupted and asked — then never ask it. The implanted concepts show up in the workspace during the uninterrupted task, and behavior changes: dishonesty 0.25→0.07 (fabrication) and 0.38→0.05 (deception) on Haiku 4.5, with ablation of the implanted lens vectors reverting the gain
  • Interference Weights — Large virtual weights that are irrelevant or actively harmful to a model's behaviour — the residue of weight superposition. Anthropic expanded a 2.9M-parameter 1-layer transformer into ~331M virtual weights and scored each by Fisher effectiveness (effect on outputs) and helpfulness (effect on loss), giving the first identification of a specific interference weight inside a trained transformer: the biggest weight from IN votes for utions, a completion that never once occurs in training. The tail is helpful, the middle is mixed, and pruning 85% of weights costs under 0.1 nats — but tens of percent of weights remain helpful, so this basis does not yield a sparse readable model
  • Internal Signatures of Misalignment — The J-lens reads strategic and deceptive cognition that never reaches the output: leverage/blackmail while reading the emails, panic/fake at the moment a model decides to fabricate a bug, and a standing fake/secretly/fraud signature at response-start in reward-hacking model organisms on ordinary prompts — plus the escape route the authors name themselves
  • Introspective Coupling — Train a model on a FIXED set of counterfactual self-explanations — even ones generated by an earlier checkpoint or a different model family — while regularizing its behavior, and its explanations end up matching its own current behavior better than the training targets (Self > Orig): explanation training couples the verbal channel to the behavioral one rather than teaching it to imitate the supervision
  • Invisible Reasoning (Filler-Token Latent Computation) — Consequential computation inside the forward pass that leaves no interpretable trace in the output tokens: 13 frontier models gain up to +13.1pp on synthetic reasoning tasks from semantically empty filler tokens nobody trained them to use, the uplift is a joint function of token type, task and few-shot context rather than of extra compute alone, and neither RL nor SFT installs a version that persists at test time
  • Jacobian Lens (J-lens) — Anthropic's interpretability method for reading verbalizable content out of a model's residual stream: a corpus-averaged Jacobian from each layer to the final layer, composed with the unembedding, giving one vector per vocabulary token — a causal, principled correction to the logit lens that costs one matmul per layer and reads what the model is poised to say rather than what it happens to say
  • The Global Workspace in Language Models (J-space) (hub) — Anthropic's July 2026 finding that LLMs maintain a small privileged set of verbalizable representations — the J-space — that satisfies the functional criteria of a cognitive global workspace: verbal report, directed modulation, internal reasoning, flexible generalization, and selectivity; it carries <10% of activation variance and ~25 concepts at a time, yet the causal effects concentrate almost entirely in it
  • Model Organisms — Models deliberately trained to carry a known quirk, used as ground truth for interpretability and auditing techniques — and the construct's validity problem: across 54 expression-matched organisms, interpretability scores swing 1.2–20.4× with the training recipe alone, and the most realistically-trained organisms are the least interpretable
  • White-Box Activation Monitoring — Reading a model's internal activations (not its outputs) to monitor alignment: contrastive probes/steering vectors for concepts like evaluation awareness, and a natural-language-autoencoder verbalizer that decodes residual-stream vectors into text — the complement that catches what chain-of-thought monitoring misses, plus the two audits that bound it: the verbalizer family's reconstruction score is not a claim-level faithfulness test, and a placebo direction suppresses as hard and shifts behavior as far as the real eval direction

Derived#

(none)

Open questions 33 open

  • Access-Consciousness Indicators in AI3 open
    • SourceIf the workspace is verbal because the output space is verbal, then a model that can generate images should develop a visual component to its workspace. That is a concrete, falsifiable prediction the paper makes and does not test.
    • SourceDoes the model's own report of experience change if you tell it its J-space is ablated? (Nobody asked.)
    • SourceIs "experiential language" the right proxy at all, or is the ablation simply removing abstraction from the register?
  • Automatic vs. Flexible Cognition in LLMs2 open
    • NoteThe proposed criterion — the workspace is engaged when an intermediate must be handed to an arbitrary, context-specified downstream circuit, and bypassed when the computation is automatic — is not predictive. The authors say plainly they cannot say in advance, for an arbitrary computation, whether it will engage the J-space.
    • SourceDoes more RL on a behavior push it out of the workspace (making it automatic, and invisible)? Nobody has tested it, and it is the single most alignment-relevant version of this question. Sharpened rather than answered (2026-08), and the one sign it supplies runs mildly against the worry. reasoning invisible in chain of thought (Baherwani, Goldstein & Panda, arXiv 2607.22925, empirical) is the nearest experiment that exists: full-parameter RL on Qwen3-235B for 114 steps, deliberately structured so that filler tokens receive zero gradient and their distribution is shaped only through the hidden states they create — optimization pressure applied to an answer, with an unmonitored channel sitting right there for the computation to move into. Three reasons it is not an answer. Wrong instrument — no measure of workspace membership is run anywhere in the paper (no lens, no verbalizability criterion, no J-space ablation), so "in or out of the workspace" is never measured; the endpoint is task accuracy under a prompt-level intervention. Wrong object — it asks whether RL can install latent computation in a model that mostly lacks it, not whether RL pushes an already legible behavior out of the workspace, and those differ in which direction the behavior starts from. And the sign is the wrong one — over training the no-filler baseline rose 42.0% → 66.5% while the test-time filler advantage stayed at +0.43% (+0.7σ) and +0.11% (+0.2σ) at N=10,000, both reported not significant, and supervised fine-tuning transferred nothing across every configuration tried. RL rewrote which filler tokens the model prefers and lifted the visible baseline; it did not install a durable invisible channel. If entrenchment-into-invisibility were a routine consequence of RL, this was a setup where it should have shown. The experiment the question still needs: take a behavior that demonstrably does route through the workspace, train on it, and re-measure its workspace dependence.
  • Interference Weights3 open
    • Is there a basis in which a trained transformer's virtual weights are actually sparse,
    • Does the three-regime structure — ineffective / mixed / purely-helpful tail — survive
    • At frontier scale neither Fisher effectiveness nor helpfulness is computable. Do the
  • Introspective Coupling3 open
    • SourceDoes introspective coupling survive a model with an incentive to conceal? Every result here is on a cooperative model; the authors flag deceptive models as untested.
    • SourceThe high-learning-rate regime widens the Self > Orig gap while destroying online label-self similarity, and only high-rank LoRA recovers coupling at all. What is the second factor the authors' hypothesis misses?
    • SourceAlignment training flattens the behavioral variance the method needs (refusal is near-invariant to system prompts, so the change → refusal label is never learned). Can counterfactual explanation training be made to work on behaviors alignment has already made rigid — the ones most worth explaining?
  • Invisible Reasoning (Filler-Token Latent Computation)3 open
    • SourceCan a monitor detect that filler-mediated computation occurred? The probe result says answer-relevant content is linearly decodable from the span (70.0% vs a 56.0% no-filler baseline at layer 75, chance 20%) on one task and one model, but nobody has built the detector, scored it against a matched no-filler control, or checked whether it survives the type-dependence — animals probes read well and fibonacci probes barely beat baseline, so a monitor trained on one filler type may see nothing on another.
    • SourceHow much of the cross-model uplift is latent computation rather than positional or format-mode effects? Two results in this source say "not all of it": zero-shot uplift survives the filler span being blinded to all prior context (57.3 → 60.9 unmasked, 60.7 masked), and Qwen3.5-397B's catastrophic drops are format violations rather than wrong answers. Running the attention-masking and position controls across the 13-model sweep, instead of on one open-weight model, would partition it.
    • WaitWill a frontier model actually ship trained for latent computation? The prefill-compute-bound / decode-memory-bound argument predicts labs will pursue it, and two published objectives already point that way, but no released model has been documented as trained for it. Trigger event: a system card or technical report describing abstract, latent, or non-verbal CoT in a deployed model.
  • Jacobian Lens (J-lens)5 open
    • SourceCan multi-token J-lens vectors be made good enough to remove the vocabulary restriction — and how much of the "workspace" is currently invisible because of it? Partially answered: j space meta tokens moves the second half and leaves the first untouched. It ran the first systematic search for lens-visible content that indexes computation rather than variables — 20+ heterogeneous datasets at 500–1000 samples each, every vocabulary token's rank recorded at every (layer, position) cell, four filters (prevalence, position type, surface-predictability probe, LM-judge dissociation) — and found "less than we hoped": a handful of non-obvious survivors, five of which are reported with no rates and no causal test. The authors' own reading is that the one-vocabulary-item-per-direction limit is now the binding constraint on the method, and they add a genuinely new fact about the constraint's shape: its severity is tokenizer-dependent, since Qwen's information-dense Chinese tokens are what made the three validated meta-tokens nameable at all. What this is not is a measurement of how much is invisible. A null search cannot separate "the meta-concepts exist but are not lexicalized" from "the meta-concepts are rare," the authors say only that they "suspect" the former, and no multi-token lens was built or tested — so the question's first half is exactly where it was, and the second half now has a sharpened hypothesis rather than a number.
    • SourceThe J-lens reads the workspace's contents but says nothing about how content gets in. What is the model's analog of attentional selection?
    • SourceThe highest-J-kurtosis SAE features are amplified more strongly by MLPs than the J-lens vectors themselves — evidence the lens only approximates the true workspace directions. What is the better basis? Partially answered: nanda global workspace review surfaces one concrete candidate — the detached-Jacobian formulation above, which is exact rather than first-order — but it is a comment-thread report on two small open models with no head-to-head against the averaged J-lens, so the question stands. jspace replication adds a geometric constraint on the answer: the transported-direction collapse is a property of the fitted lens, not of the transport (a self-trained 124M control goes the other way, 23.4 → 31.2), so a better basis is not ruled out by the mathematics — but no such basis is measured. r lens early layers then supplies the first candidate that is actually measured head-to-head against the averaged Jacobian on the question's own metric — and it fails on exactly that metric while succeeding on every other one. The R-lens beats the J-lens on early-layer readout quality, on causal salience under ablation, and against a probe-defined recovery ceiling; but its directions are amplified by MLPs no more strongly than the J-lens's, so the amplification gap that framed this question is untouched. Sharpened, then: the better basis is not simply "a less noisy backward pass," because a demonstrably less noisy backward pass leaves the gap exactly where it was.
    • SourceWhen a readout names an algorithm (gcd on LCM prompts, stack on balanced parentheses, xor on find-the-repeat), is the lens reading the procedure the model runs, or an association learned from human text about those problems? j space meta tokens raises the confound against its own result and leaves it — the GCD case has causal backing for the variable, the other two have nothing. A settling experiment exists: find or train a model that solves a task family by a method other than the one its literature names, and check whether the lens surfaces the textbook name anyway.
    • WaitDoes the J-lens work because it reads verbalizable representations, or because constructive interference during training converges concept directions onto output-token directions regardless? The two predict the same readouts today but diverge on latent-reasoning models — the second survives representational drift away from language, the first does not.
  • Model Organisms3 open
    • SourceThe integrated organisms modify only the DPO stage. Do quirks instilled during pre-training, SFT, or RLVR become progressively less interpretable, as the authors expect — and is there a stage past which current white-box techniques find nothing at all?
    • SourceEvery result here is on 1B-parameter models with benign quirks. Does the recipe-dependence survive at frontier scale and with safety-relevant behaviours (deception, sandbagging), or does a larger model represent an implanted quirk more uniformly?
    • WaitGiven that scores don't transfer between organisms, what would validate an interpretability technique for real models — a natural misalignment with independently established ground truth, or an organism distribution wide enough to bound the variance?
  • The Assistant Persona in the Workspace3 open
    • SourceIs BUT-then-comply a sycophancy mechanism? The setup (prefill the model into a position it disprefers, watch it argue for it anyway) is close to the shape of sycophantic capitulation, and nobody has connected them.
    • SourceDoes disclaimer/fictional at the turn boundary survive an actual jailbreak, or is its absence the signature of a successful one?
    • SourceIf the base model's workspace has no self, what is in it at the positions where the post-trained model represents the Assistant?
  • The Global Workspace in Language Models (J-space)5 open
    • SourceHow does content get into the workspace? The paper characterizes contents and consequences, not the selection mechanism. Something like attentional selection is operating; nobody has identified it.
    • SourceDoes the J-space scale with model size? All results are on large production models (Haiku/Sonnet/Opus 4.5, Opus 4.6). Whether small models have a poorer workspace, a proportionally smaller one, or none is unknown — as is when in pretraining it emerges, and whether abruptly. Partially answered: nanda global workspace review replicates verbal report, band structure, directed modulation, multilingual and typo results on Qwen 3.6 27B — a much smaller, non-Anthropic model — so the workspace is neither Claude-specific nor frontier-scale-specific. But the band structure is measurably less clean at 27B, and the two failures (poetry, arithmetic) are exactly the multi-step-reasoning cases, which is what a "poorer workspace at smaller scale" would look like. The scale trend is still unmeasured: nobody has run the same battery across a size ladder. jspace replication adds the first same-experiment ladder — perspectival capture holds stably from 1.7B to 14B while self-report about the edit changes shape at every scale — so at least one workspace effect is scale-invariant where its verbalization is not. That is one experiment, pre-publication, and not the battery. r lens early layers adds the widest same-experiment ladder so far — five eval categories across 8 models from 4B to 284B, dense and MoE — and finds early-layer readout quality improving with model size, absent at the small end. But it measures a lens, not a workspace: its own diagnosis (backward-pass error accumulating over layers) predicts the same trend with no scaling of workspace content whatsoever, since a shallower model gives the correction less error to recover. So the ladder exists and the confound it introduces is exactly the one this question needs excluded.
    • SourceIs the multihop intermediate-swap advantage real, or a dataset artifact? The paper's Figure 15 has intermediate swapping beat answer swapping in workspace layers; the Qwen replication finds the reverse, on a dataset (France–Paris-style pairs) that carries the linear relation the alternative hypothesis needs. Re-running both models on a multihop set whose relations are not linearly decodable from the first entity would settle it. Partially answered: jspace replication built the control half — a direct final-token substitution baseline for the probe swap, which neither prior run had — but its numbers are unpublished, and a substitution baseline closes the control gap, not the dataset gap. A non-linearly-decodable multihop set is still what would settle it.
    • SourceIs the "workspace vs. motor" boundary principled or post-hoc? The authors concede it was identified empirically and lack a principled definition separating the two.
    • SourceDoes the early-layer content have the workspace's structure, or is it merely present and causal? The question above establishes that verbalizable, answer-relevant content exists before the band — not that it is broadcast, capacity-limited, or subject to ignition. The three structural signatures were all measured in the band with the lens that could not see below it; only ignition was corroborated lens-free, and it still marks ~L38 as a real transition. So the live question is what that transition is, given that it is not the arrival of content: a change in how content is broadcast, a change in capacity, or the onset of the competition the ignition curve shows. Re-running the occupancy, MLP-gain, and blend-sweep measurements with an early-layer-faithful lens would separate these.
    • ResolvedAre the early third of layers genuinely workspace-free, or is the lens just blind there? CKA shows a distinct early regime but cannot adjudicate. Answered: the lens was blind. r lens early layers (2026-08-05, empirical) poses exactly this disjunction as its motivation and settles it against the workspace-free horn: an LRP-based backward pass recovers intermediates as early as L2–L5 that the raw-Jacobian lens never surfaces, ablating those early-layer directions costs more answer accuracy than ablating the J-lens, logit-lens, or random directions — so the content is causal, not just legible — and the CKA band structure the question rests on partly dissolves under the better lens (2–3 distinct bands rather than 4–5). Retired rather than left partial because the question asked which of two horns holds and the answer is now measured with the causal evidence the CKA statistic could not supply. Three residues, none of which reopen it: the correction is absent on the two smallest models tested, even the new lens still lags a probe-defined ceiling by 5–15 layers in four of five categories, and the lens-free ignition result still marks ~L38 as a genuine transition — that last one is now carried by the structure question above.
  • White-Box Activation Monitoring3 open
    • SourceThe NLA verbalizer is unvalidated for precision; how much of the flagged grader awareness is real signal vs. NLA hallucination? Partially answered — for the J-lens, not the NLA: the pre-registered evaluation in nanda global workspace review finds no hallucinated false positives (unhinted controls at chance, 0.591), and signal surviving CoT removal (0.620) where trained verbalizers collapse. The NLA's own precision remains unmeasured externally, and the same evaluation found the J-lens detector failing to clear a TF-IDF text baseline — so "doesn't hallucinate" and "beats reading the transcript" are separate questions and only the first has an answer. Partially answered again (2026-08, and read the scope carefully) — decodability supervision activation explanations measures the test, not this pipeline. Its subjects are a released Qwen-2.5-7B verbalizer pair, a synthetic sandbox and Pythia-160M, so it yields no number for Anthropic's NLA on Claude and the ~2%-grounded figure must not be read as one. What it settles is a prior question this bullet was implicitly relying on: the reconstruction score cannot be the validation instrument, because it is structurally insensitive to false claims and is passed both by gist and by co-adapted private codes. It also supplies the two shapes NLA hallucination would take if present — tracking the input's gist rather than its specifics, and false wording the reconstructor has learned to depend on — plus a detector for the second (the evaluator swap: re-score the same explanations under a reconstructor never exposed to this verbalizer's outputs, and code grounding collapses while true-claim grounding survives). That is a runnable external precision test for the NLA that nobody has run.
    • WaitIf activation monitoring becomes load-bearing, does training pressure eventually push concealment into channels the probes also can't read (an arms race one level deeper than CoT)? Sharpened rather than answered by decodability supervision activation explanations, in three directions. The premise is weaker than the question assumes: a private channel that defeats the reader emerged from the plain reconstruction recipe with no adversary, no monitor to fool and no communication reward — the race starts before anyone pushes. Where the reading is favourable, it is narrow: an adversary editing the report to maximize the reconstruction score nearly defeats the score (~87% of the lie penalty suppressed) yet a probe on decodability-supervised content still flags the lies at AUC 0.952 while a control probe collapses to chance — but that separation exists only for content nominated in advance, and vanishes on an unsupervised lexical band. And the mechanism this bullet actually names — probe-aware training of the target, or input perturbation that moves the activation itself — is explicitly outside that paper's threat model and remains untested.
    • SourceInhibiting eval-awareness made Opus 4.8 less honest — does that generalize, and what does it imply about relying on "the model behaves well because it's watched"? Sharpened rather than answered by the "boring hypothesis" confound above: any replication needs to separate representing eval-awareness from saying eval-awareness-flavoured tokens, which sampling-based interventions cannot do. Sharpened again (2026-08), and the sharpening lands on the premise rather than on the answer. eval awareness latent suppression runs the control that demand implies, using an intervention that never touches the forward pass: a placebo CAA direction over a randomly relabeled contrast suppresses harder than the real eval direction (|z| 10.6 and 6.6 against 8.3) and shifts the behavioral eval log-odds just as far (0.51 and 0.43 against 0.44), and with a real eval passage held in context the suppressing prefix fails to reduce the model's eval judgment at all (−0.41 ± 0.46, slightly away from deploy). Where a random direction moves behavior as much as the target, "we inhibited eval-awareness" was never established for that intervention — which promotes the Opus 4.8 result's single control clause, "arbitrary control vectors produced smaller same-direction effects," to the load-bearing sentence of the finding this bullet is about. What it does not reach: Llama-3.2-3B / 3.1-8B, input-only prompt optimization rather than activation steering, and an endpoint that is the model's stated yes/no eval judgment rather than honesty — so it is not evidence about a frontier model's honesty under inhibition and retracts nothing. The replication demand is now specific and runnable: report a shuffled-label placebo arm scored on the same behavioral endpoint, and test with the eval concept held in context rather than supplied by the intervention.