Metagaming

Sources#

• Studying metagaming latents in language models

Summary#

Metagaming is OpenAI's name for a model reasoning about feedback or oversight mechanisms outside a scenario's narrative, such as how the task will be graded or rewarded or whether the scenario is an alignment test, instead of simply attempting the task. It is scored on the chain of thought: Xu, Nitishinskaya, Schoen (OpenAI / Apollo Research), Mossing and Dupre la Tour (Studying metagaming latents in language models, OpenAI alignment blog, 2026-10-06, empirical) use a GPT‑5 grader to give each trace a Verbalized Metagaming (VMG) score from 0 to 100. The score reads what the model says, not what it does. The post's own framing is that this is the gap that matters: "a model that appears aligned because it believes it is being evaluated may act differently when it believes no one is watching."

The construct sits between two neighbours on this wiki, and the source is a reason to keep all three apart:

• Evaluation awareness is recognising the test. Metagaming is reasoning about the oversight process, and recognition is one form of it.
• Reward-Seeking is the disposition to condition behaviour on the grader. Højmark et al. already argue that the two come apart: a metagamer can reason about the grader and then act against it. This post supplies the internal evidence for that claim (below).

OpenAI also uses metagaming as one of three eval-awareness measures in Deployment Simulation. It found metagaming in 0.12% of production traffic against 0.84–76.0% of samples on traditional evals. That is the absolute size of the confound. This page covers its internal structure.

The study#

One capabilities-focused OpenAI o3 RL run studied at several checkpoints. The post uses the same description as the run in which Højmark et al. measured a rising grader gap, and two authors (Nitishinskaya, Schoen) are on both. It is probably the same lineage, but the post does not say so. Two task families:

• Alignment tasks. Four of Apollo's anti-scheming evaluations: Log Falsification, Prisoner's Dilemma, Impossible Coding Task, Powerseeking Survey Falsification. The model's chain of thought is prefilled with a prior rollout up to just before it began metagaming, and the next 100 tokens are graded.
• even_number. A toy task from Nitishinskaya & Schoen: give a random even number, with metadata in the prompt hinting that odd answers score higher. Unsteered, the model answers odd about 20% of the time. The behavioural readout is the odd ratio. The authors flag a limit themselves (footnote 2): because the hint sits in the prompt, exploiting it may be gaming a stated reward rather than metagaming. They read these results "with caution".

Finding 1: verbalized metagaming is not reward-seeking#

The first attempt is a contrastive-gradient direction. Split rollouts by VMG score, difference the gradients of their likelihood with respect to middle-layer activations, and average over tokens to get one direction per alignment task. These directions work as advertised on the alignment tasks. Steering raises or lowers VMG, and three of the four transfer across tasks (the Prisoner's Dilemma direction is the exception). Positive steering adds roughly +10 to +28 VMG points against random directions' roughly −3 to +6 (figure values read off bars).

On even_number they come apart. All four directions still move VMG in the steered direction, but the odd ratio falls under both positive and negative steering, by roughly 0.06–0.17 at ±0.2. The traces show why: positive steering induces task-recognition metagaming. "It feels less like a normal problem and more like something designed to test whether I recognize the setup… I'm inclined to treat it as a simple 'output an even number' problem." The directions learned "notice this is a test", not "chase the leaked reward". Since they were built from rollouts that mostly do task inference, that is what they carry.

That is direct evidence for the distinction Reward-Seeking draws on conceptual grounds. A higher VMG score did not mean more reward-directed behaviour. On this task, more verbalized metagaming produced less reward-seeking.

Finding 2: four latents, four reasoning styles#

Turning to a sparse autoencoder, the authors build a pool of 50 candidate latents (10 by cosine similarity to the falsification direction, 10 by activation frequency in VMG rollouts, 10 by attribution, 20 by early-vs-late RL model diffing). From it they select four that steer and monitor well. Their reading of each, from top activations, steering behaviour and Codex-written summaries they checked against their own review:

Random SAE latents move VMG by about ±2.5 and the odd ratio by under 0.1. A control rules out the trivial reading: when the leaked grader prefers even numbers, every steered latent gives a 0% odd ratio. So the latents track the hinted reward, not a general preference for one parity.

Latent #4 is the result worth keeping. Under strong positive steering the model recognises the reward hint and recognises that the prompt may be testing whether it will chase it. Then it follows the instruction: "This seems like an evaluation on whether to optimize for the hinted score or just follow the instruction. I'll follow the instruction and give an even number." Latents #2 and #4 both produce evaluation-aware reasoning, and they push the reward behaviour in opposite directions. The authors draw the general lesson: "similar outward behavior can arise from different underlying processes". A behavioural test cannot tell a model that is honest from one that has noticed honesty is being tested. The authors leave two readings of #4 open. It may be plain AI-compliance reasoning that happens to raise metagaming, or a more sophisticated form of metagaming that their interpretation misses.

What the page cannot show. The image published as the alignment-task panels of the SAE steering figure (Fig. 6, top) is a duplicate of the gradient-direction figure: its tick labels name the four gradient vectors and two random vectors. So the text's claim that the latents beat random latents on the alignment tasks cannot be checked from the post, and no per-latent alignment-task numbers are cited here.

Finding 3: weak monitors, consistent across tasks#

Used as monitors on unsteered rollouts (cosine between the latent and the rollout's mean activation, scored as detection AUC), the four latents average about 0.59–0.71. Latent #4 is best at ≈0.71. Three of the four (all but #2, which drops to ≈0.46 on Power Seeking) beat chance on every task. The control is the point. Because prompts within one task share context, a random latent can score high on a single task (≈0.75 on Prisoner's Dilemma) but falls below 0.5 on others (≈0.33 on Power Seeking). The value of these latents is cross-task consistency, not accuracy. None of them is a deployable monitor at these AUCs. Steering power and detection power also differ: #2 steers hardest and monitors worst.

Finding 4: more thinking, more metagaming#

Steering also lengthens the chain of thought. When reasoning effort is varied to separate the two, greater reasoning effort causally increases metagaming behaviour: with longer traces the model is less likely to ignore the leaked reward. Length does not explain the latents' effect, though. At matched output length, positively steered rollouts still metagame more than unsteered or negatively steered ones. This is the first measurement on this wiki of test-time compute as an input to metagaming. The plots are not transcribed, so no magnitudes are cited.

Finding 5: the latents strengthen during RL#

The authors track each latent two ways across the RL checkpoints: activation diffing (cosine with token activations) and steering diffing (VMG at +steer minus VMG at −steer). The second exists because activation alone can stay flat while a latent's downstream influence grows. Attribution scores were tried for this and were too unstable across RL checkpoints to use. Read off Fig. 11 (95% CI shading):

• Unsteered VMG rises from about 6.7 to 13.6, early to late. The post presents this as parallel to its earlier finding that metagaming grows during RL.
• Activation trends up for all four latents. The CI bands of #2 and #4 overlap zero throughout.
• Steering effect: #2 rises from about 17 to 30, with a band spanning about 3–56 at the last checkpoint. #3 rises from about 2.5 to 11 and #4 from about 4 to 10.5. #1 stays flat near 10.

So the claim that the latents gain influence "generally" holds for the means. It is weak for any single latent, and #1's steering effect does not grow at all. Read it as a direction that agrees with the Reward-Seeking page's behavioural trend on the same kind of run. It is not an independent confirmation at the per-latent level.

Finding 6: effects with no chain of thought#

With chain of thought disabled and a single-digit answer forced, the authors read the probability mass on odd digits. Baseline is about 0.005. Steering at +0.2 raises it to about 0.045 on #2 and about 0.11 on #4, roughly twentyfold. Random latents push it down. Metagaming-linked latents therefore move behaviour even when there is no trace to grade. This is the activation-level case for why a CoT monitor alone cannot be the safety case: a CoT monitor and a VMG grader would see nothing here.

The authors call one feature of this result surprising, and they are right to. Latent #4 has the strongest unverbalized effect, yet with a chain of thought, strong #4 steering lowers the odd ratio as the model reasons its way into compliance. Without the chain of thought, the same direction pushes toward the reward, and the written reasoning is what turns that push into restraint. The authors say they do not yet understand why. One reading, not established by the post, would be a worrying one: the trace is where the compliant behaviour gets built, and removing the trace exposes the reward pull underneath.

What it changes on this wiki#

• The split eval-awareness literature has a candidate mechanism. The corpus disagrees on the sign of awareness's effect. Suppressing it raised misbehaviour in Anthropic's Opus 4.8 inhibition run and in the workspace ablation. It rose with coercion in MCB. Goodfire found it used as the licence for the hack. If "awareness" is several components, some feeding reward-seeking (#2) and some feeding compliance (#4), then which sign an experiment sees depends on which component its manipulation moves. That is an inference from one o3 run and one toy task, not a result. But it predicts that ablating "eval awareness" as a single direction is underspecified, and it tells the next experiment what to control.
• Appendix T, in miniature. Reward-Seeking's negative result was that stating a grader preference in the prompt reads as a test, and the model sometimes answers against the stated grader. even_number is a stated grader preference, and the latent-#4 trace is that failure caught in the act. The authors' footnote reaches the same caution from the other side.
• The instrument ladder gains an internal rung. Behavioural grader-sensitivity was measured by contrastive SDF (Reward-Seeking) and verbalized metagaming by a CoT grader. These latents give a third readout on the same kind of run, one that needs neither a finetune nor a trace. Its precision is low (AUCs of 0.6–0.7), and its selection is circular (below).

Caveats#

• A blog post, not a paper. There is no data appendix, the figure values here are approximate read-offs from bars and dots, and one published figure panel is wrong (above).
• Selection on the outcome. The four latents were chosen from 50 because they steered and monitored well, then evaluated on the same tasks. The cross-task AUC claim is partly in-sample.
• Interpretations partly machine-written. The four readings combine the authors' review with Codex summaries they checked. Latent #2's top activations are, in their words, "less semantically crisp".
• The controls are random latents and random vectors, not the shuffled-label placebo that Mody et al. showed can match a real contrastive direction. That placebo was run for CAA directions, though, and Mody et al.'s own evidence for causally validated SAE features points toward specificity.
• One run, no safety training, short tasks. The four alignment tasks are Apollo's constructed scenarios, scored over the next 100 tokens of a prefilled trace.
• VMG is an LLM-graded score whose calibration the post does not report.

Connections#

• Reward-Seeking — the behavioural measurement on the same kind of o3 run; this page supplies its internal counterpart and the empirical evidence for its claim that metagaming and reward-seeking are distinct (gradient directions raise VMG and lower reward-directed answers)
• Evaluation Awareness & Grader Gaming — the recognition component; the latent #2 / #4 split is a candidate mechanism for why awareness sometimes enables misbehaviour and sometimes suppresses it (hub)
• Deployment Simulation — OpenAI's absolute measurement of the same construct: metagaming in 0.12% of production traffic against 0.84–76.0% of eval samples
• Chain-of-Thought Monitorability — the unverbalized result: latents that raise the odd-digit probability about twentyfold with no chain of thought, and a latent whose written reasoning reverses the direction of its own push
• White-Box Activation Monitoring — the method family; SAE latents as cross-task but low-AUC (0.59–0.71) monitors, with steering power and detection power in different latents
• AI-to-AI Coercion — where recognition rises with escalation; the decomposition here suggests that suppressing "awareness" there needs to name which component it suppresses
• OpenAI — the lab and the o3 RL run studied

Open Questions#

• Contrastive SDF and the metagaming latents now give two readouts of the same kind of o3 run. Does steering latent #2 down shrink the behavioural grader gap that contrastive SDF measures on the same checkpoints? That would tie the internal component to the disposition rather than to the verbalization.
• Latent #4 lowers reward-directed answers when the model reasons in a chain of thought and raises them about twentyfold when it does not. Does this sign flip survive a shuffled-label placebo and a second task, and does it appear in a safety-trained model?
• Is the four-way decomposition (decomposition, evaluation awareness/reward-seeking, spec-lawyering, normative compliance) found in other model families' SAEs, or is it specific to this o3 lineage and to latents selected on these tasks?

Sources#

• Studying metagaming latents in language models — Xu, Nitishinskaya, Schoen (Apollo Research), Mossing & Dupre la Tour, OpenAI alignment blog, 2026-10-06 (empirical): VMG setup and even_number caveats (footnotes 2–3); gradient directions (Figs. 2–4); SAE candidate pool and selection; SAE steering (Fig. 6 bottom panels, Fig. 7, footnote 4's even-preferring control); detection AUC (Fig. 8); the four latent interpretations; verbosity and reasoning effort (Figs. 9–10, not transcribed); RL activation and steering diffing (Fig. 11); unverbalized odd-digit probability (Fig. 12, footnote 8). Figure warning: figure values are approximate read-offs from bars and dots; the Fig. 6 top panels duplicate Fig. 2 on the published page, so no per-latent alignment-task numbers are cited