The Verifiability Thesis

Sources#

• Andrej Karpathy: From Vibe Coding to Agentic Engineering
• CS329A Self-Improving AI Agents — Part 1: Course Overview
• CS329A Self-Improving AI Agents — Part 2: Test-Time Compute Scaling
• CS329A Self-Improving AI Agents — Part 3: Robust Verification
• CS329A Self-Improving AI Agents — Part 9: Future Research Areas

Summary#

Andrej Karpathy's organizing claim for what AI automates and when: traditional computers automate what you can specify in code; LLMs automate what you can verify. Because frontier labs train models in giant reinforcement-learning environments with verification rewards, capability peaks sharply in verifiable domains (math, code) and stays rough elsewhere — producing Jagged Intelligence (Ghosts, Not Animals). The practical decomposition: a capability shows up if it is verifiable and the labs cared enough to build environments / include the data. Verifiability is therefore both an explanation of today's jaggedness and a strategy lever — if you can construct verification, you can pull the RL/fine-tuning lever yourself.

The core analogy#

Traditional computers can automate what you can specify in code. This latest round of LLMs can automate what you can verify.

RL training rewards verified outcomes, so the gradient flows hardest toward domains where correctness is checkable. Math and code are the canonical winners — and not coincidentally the domains where AI-Driven Formal Proof Search (Lean + compiler) and agentic coding (tests + CI) are most powerful. The compiler/test is the verifier; the verifier is the reward signal.

"Verifiable + labs care"#

Verifiability alone isn't sufficient — the labs also choose what enters the mix:

• The chess anecdote. GPT-3.5→GPT-4 chess improved far more than the general capability curve predicted, because "a huge amount of chess data made it into the pre-training set." Someone decided to add it; the capability spiked. You are "slightly at the mercy of whatever the labs happen to put into the mix."
• Implication for users. The model "has no manual." You must explore it: figure out which circuits you're in. "If you're in the circuits that were part of the RL, you fly; if you're out of the data distribution, you struggle" — and then you have to do your own fine-tuning.

The founder lever#

Karpathy's advice to founders chasing verifiable domains the labs haven't prioritized: verifiability is "technology that just works — you can pull a lever." If you can assemble diverse RL environments / examples, you can fine-tune and "get something that actually works pretty well." He coyly declines to name "one domain that's very [valuable]" — a deliberate non-answer that flags an unexploited verifiable niche as a startup opportunity. (Cross-ref the moats discussion in Seven Powers Applied to AI and Compounding Data Moat: a proprietary verification environment is a cornered resource.)

Everything is eventually verifiable#

On the flip side — "what stays automatable only from a distance?" — Karpathy argues almost everything can be made verifiable to some extent. Even soft domains like writing yield to "a council of LLM judges" producing something reasonable. So the question is how easy or hard, not whether. This is the optimistic horizon of the thesis: verifiability is a spectrum that AI keeps climbing, with LLM-judge ensembles extending the reward signal into fuzzy domains.

The same thesis, taught as the generator–verifier gap#

Stanford's CS329A arrives at Karpathy's dichotomy from the training side and gives it a different name (lecture 1, delivered 2025-09-22, practitioner-opinion). Aakanksha Chowdhery: generation is cheap and abundant — a model will happily produce "a whole bunch of nonsense or sensible set of reasoning traces" — so the binding constraint is the feedback that says which. The gap between how easily a system generates and how easily it checks is what determines whether a domain improves.

Two things this framing adds to the page:

• The bottleneck has a name on the human side too. Where mechanical verification is unavailable — the lecture's example is creative writing — "human feedback ends up becoming a bottleneck," and the domain stops improving not because models can't write but because nobody can afford to grade at scale. Karpathy's optimistic horizon above ("a council of LLM judges") is precisely the proposal to route around this, and Reference-Free Judge Over-Crediting is why routing around it is harder than it sounds.
• It gates the self-improvement flywheel, not just capability. The course's whole thesis — test-time compute manufactures verified trajectories, those become training data (Large-Scale Test-Time Compute, Recursive Self-Improvement) — runs only where the verifier is cheap. So verifiability decides not only what today's model is good at but which domains can bootstrap at all. Azalia Mirhoseini treats "robust verification is hard" as the live research problem and works on combining verifiers rather than trusting one.

The dated part (late 2025): the instructors present verifiable-domain agents — code with unit tests, maths with known answers — as the only place the loop demonstrably closes. The 2026 sources on this page argue the frontier has moved without ever removing the gate.

The gap, measured (CS329A lecture 2)#

Lecture 2 (CS329A Self-Improving AI Agents — Part 2: Test-Time Compute Scaling, Azalia Mirhoseini, delivered 2025-09-26) turns the generation–verification gap from a framing into a plotted quantity, and it is the sharpest statement of this page's thesis in the corpus because it isolates the verifier as the only moving part: the generator is frozen, and every arm differs only in how the samples are judged.

Sweep sample count and plot success rate against it:

• Majority voting plateaus at roughly 10–50 samples while coverage (the perfect-selector line) keeps climbing. The vertical distance between them is the gap.
• Reward-model rankers barely dent it. Outcome reward models, best-of-N against them, and combinations of majority voting with reward scoring all sit far below coverage.
• The gap widens with difficulty. It is larger on MATH than on GSM8K — "this gap is even more pronounced if you have harder problems."

The mechanism is the part worth carrying, because it makes the failure structural rather than a matter of a better ranker. On the hardest problems that are solved, the correct answer appears once, twice or three times in 10,000 samples (Latent Capability Overhang). Majority voting is a frequency estimator; the answers worth finding are, by construction, the rarest ones. A selector that ranks by consensus is blind exactly where the coverage lives. On GSM8K, where responses are simple, consensus works for most problems — and still fails on the hard tail.

The verifier taxonomy, ordered by how much you get for free. The lecture's ladder of what "verifiable" can mean:

1. Formal proof — check each step with a proof assistant (AI-Driven Formal Proof Search).
2. Unit tests — and the asymmetry that makes them work: "writing a unit test arguably is a much simpler task than writing the entire program."
3. Output equivalence — the "AI as a compiler" case: generate CUDA from PyTorch and check the two produce identical outputs on arbitrary inputs. The reference implementation is the verifier, and the same trick covers any language-to-language port. This is a verifier you get without writing one.
4. Model-based scoring — outcome and process reward models, LLM judges. Where the gap lives.

And the caveat the lecture attaches to its own top rungs. Asked whether reported coverage might be inflated by wrong-but-passing answers, Mirhoseini's answer is a manual check: on the maths problems, hand-inspection put true correctness somewhere around 90% — and she immediately generalizes it, "that's always a case for a failure mode if your unit tests don't have true coverage of the code." A hard verifier is a cheap verifier, not a sound one, which is the same crack Stopping Under a Noisy Verifier widens into a control problem and Agent-Generated Test Quality locates in the tests themselves. A student in the same session proposes the asymmetric route around it — verifying correctness may be too expensive while falsifying it is cheap, so filter out demonstrably-wrong answers and iterate evolutionarily — which is the shape Reference-Free Judge Over-Crediting's solve-then-compare result later argues for on measured grounds.

The gap's second component, and two routes at it (CS329A lecture 3)#

Lecture 3 (CS329A Self-Improving AI Agents — Part 3: Robust Verification, same lecturer, delivered 2025-09-29) is the course's dedicated verification lecture and it splits rung 4 above into its own literature. Two things belong on this page; the machinery lives on Process vs Outcome Reward Models (papers 1–3) and Weak-Verifier Ensembling (paper 4).

The gap is two failures, not one. Lecture 2 established that majority voting plateaus at 10–50 samples because it is a frequency estimator blind to the rare-but-correct answer. Lecture 3 adds the failure mode of a trained verifier, which is different in kind: swept against sample count, OpenAI's 2021 GSM8K verifier improves to roughly 400 samples and then declines, because at 800 candidates two near-identical solutions — one right, one wrong — are harder to separate than they were at 400. A learned selector does not plateau; it degrades. The shipped system stopped at 100 samples. So the ladder's softest rung buys about an order of magnitude more usable samples than consensus and then hits a ceiling of its own, and neither ceiling moves by sampling more. The constructive counterweight on the same slide deck: a human-labelled PRM reportedly surfaces correct solutions on problems where under 5% of samples are correct — the regime the rarity mechanism above says consensus cannot reach.

And the lecture supplies the "council of LLM judges" horizon as a built system. Karpathy's optimistic pole at the top of this page — soft domains yield to a council of judges — is what Weaver is, with two things the horizon does not specify: a quality floor that excludes members scoring badly against a small labelled set ("your verifiers should be above a certain quality to even be let in the pool"), and learned per-verifier weights rather than a vote. It reports ~40% → over 70% on hard benchmarks and an open-weight stack matching o3-mini. It also rests on an explicit verifier-independence assumption that LLM-Judge Validation's ρ = 0.66–0.97 and Reference-Free Judge Over-Crediting's Proposition 2 both measure and contradict — so the council is buildable and the reason it works is not the reason its authors give. The open question below already carries the bound; Weaver is the strongest existence proof on the other side of it, and it is a practitioner-opinion one.

The one thing on this page that goes around the gap rather than at it: fusion. Coverage bounds what a selector can achieve, and every arm above is a selector. Hand all k samples to a model and ask it to synthesize one answer, and it beats the oracle — see Inference-Time Architecture Search.

The axis the ladder was missing: a verifier too slow to sit in the loop (CS329A lecture 9)#

The course's closing lecture (CS329A Self-Improving AI Agents — Part 9: Future Research Areas, both instructors, delivered 2025-12-05, practitioner-opinion) is asked the obvious question — what are the applications that don't fall into verifiable problems? — and Mirhoseini's answer reorganizes this page's ladder around a coordinate it does not have. Her examples are scientific discovery, chip design, and wet-lab chemistry, and none of them lacks a verifier:

running a very slow simulation in chip design that takes like a few days to run to collect one kind of reward signal, or running this chemical experiment that you actually need to go to a wet lab … in RL fine-tuning, or in test time scaling, we need these verifiers to be almost instant … maybe we can wait minutes, maybe 1 hour, but if in this RL training we need like hundreds or thousands of steps of iteration, we can't wait like days.

A chip simulation is a perfect verifier and a useless one. The ladder above orders verifiers by how much you get for free; this adds an orthogonal cut — latency against the number of iterations the training loop needs — and it is the one that decides whether a domain can bootstrap. Multiply a days-long reward by thousands of RL steps and the domain is unverifiable in practice while being fully verifiable in principle. That is a different failure from creative writing's, and this page had been filing both under one heading.

Three consequences the lecture states:

• The workaround is a surrogate: collect the expensive simulator's outcomes offline, train a reward model to predict them, and put the prediction in the loop instead of the simulator. Its bound is the honest one — "the generality of the reward model is a function of how much data you have," and an inaccurate surrogate is a reward-hacking surface rather than a verifier. This is the learned-verifier literature deployed where there is no cheap ground truth to train it against, which inverts that literature's usual premise.
• The genuinely non-verifiable residue is subjectivity, not slowness. Creativity and creative writing survive as the hard case, and the lecture's framing is that a reward function can always be modelled for them — at which point "the RL or the agent can do reward hacking if the model is slightly off base." Same conclusion the 2026 measurements reach; here it is stated as an expectation.
• Verifiability is a property of a task decomposition, not of a domain. Chowdhery's KernelBench refinement is the sharpest version: compiler output and execution give you a free verifier for an individual GPU kernel, but reading a performance profile across a large concatenated program does not, so the working practice is to break the problem into parts the model can verify and hand it a reference knowledge base for the rest. The gradient runs inside one "verifiable" domain, which is what the page's binary reading of Karpathy's dichotomy obscures.

Connections#

• Andrej Karpathy — author of the thesis (his verifiability writing)
• CS329A: Self-Improving AI Agents (Stanford) — the same dichotomy taught as the generator–verifier gap, and made the gate on the course's self-improvement flywheel
• Aakanksha Chowdhery / Azalia Mirhoseini — the instructors who state it; Mirhoseini's line of work is on combining verifiers rather than trusting one
• Jagged Intelligence (Ghosts, Not Animals) — verifiability is the cause; jaggedness is the symptom
• AI-Driven Formal Proof Search — the purest instance: Lean's compiler is the perfect verifier, which is why DeepMind's agents resolve open math problems
• Vibe Coding vs. Agentic Engineering — the discipline's hiring test ("red-team can't break it") is verifiability operationalized
• Evals as Product Spec — Cat Wu's "ten great evals" is the product-side mirror: encoding what "verified/done" means for an AI feature
• Verification as the New Bottleneck — Fiona Fung: once coding is cheap, verification (not generation) is the scarce resource
• Scale-Dependent Prompt Sensitivity — verifiable-domain RL is part of why bigger models don't uniformly win on every benchmark
• The Bitter Lesson — RL-at-scale in verifiable environments is the general method outrunning hand-crafted heuristics
• Client-Side Agent Optimization — fine-tuning on your own RL environments is the heaviest "pull the lever" version of the optimization story
• Compounding Data Moat — a proprietary verification environment is a defensible cornered resource
• Reference-Free Judge Over-Crediting — the council run as an actual reward and audited against held-out ground truth. It supports the thesis's dichotomy and refutes its ensemble: a verdict that commits an answer independently of the candidate inherits a judge-side ceiling and holds under optimization (false positives 0.012, discrimination 0.96), while a council of reference-free judges scoring shown candidates does not — the strictest three-family unanimous-accept rule still passes 55% of self-play-manufactured wrong answers, its discrimination collapses 0.31 → 0.09, and training against it is worse than training against one judge. Proposition 2 gives the reason: every monotone aggregation rule thresholds the same latent plausibility axis, so adding judges cannot reject a region all of them accept. Verifiability is still a spectrum you can climb — but the rung is solve-then-compare, not more raters
• Process vs Outcome Reward Models — rung 4 opened up: outcome vs process supervision, the four-year arc from GSM8K to Math-Shepherd, and the second component of the gap (a learned selector's precision decays past a few hundred candidates)
• Weak-Verifier Ensembling — the council of LLM judges as a built system, with a quality floor and learned weights, and the independence assumption the wiki's measurements contradict
• The Data Wall and the Validation Commons Are One Supply Constraint — this thesis promoted from a claim about capability to a claim about supply, on both sides at once. The ladder plus the verifier-latency axis is what rations synthetic training data (so the data wall converts into this friction rather than demoting into compute), and it is also why the Stockfish threshold arrives first exactly where human validators were never load-bearing — one variable, opposite signs, which is what dampens the uneven-arrival worry the labour-side literature raises
• Oversight When the Signals Give Out: the Activation Fallback and the Taste Reward — stress-tests the "council of LLM judges" horizon against the judge-validation cluster: reference-free judges over-credit and are gameable, so the council extends the reward signal at the cost of grounding

Open Questions#

• Where's the boundary of "council of LLM judges" reliability — does it hold for genuinely contested value judgments, or only for quality/coherence? Partially answered (2026-08-04) by Zhou (2026), and it inverts the question's premise. The question assumes the council is safe on the easy end and asks how far up it holds; the measurement says it fails on the easiest end — objective correctness on grade-school math — once anything optimizes against it. Three cross-family judges accepting only unanimously still pass 55% of manufactured wrong answers, and Proposition 2 shows no monotone rule over a shared plausibility signal can do better. Two further findings sharpen where the boundary actually sits. The council is fine as a static rater and fails as a reward: the same judges hold usable discrimination (0.21–0.38) before optimization and collapse to 0.05–0.17 after, so the binding variable is optimization pressure, not the contestedness of the judgment. And reference-free verdicts track prompt framing rather than correctness — with unit-test ground truth held fixed, Llama's gap@16 swings −0.106 under a strict instruction to +0.722 under a lenient one, so on the fuzzy end there may be no stable operating point to have a boundary about. And the council's headroom is small before any of that. Yang et al. (2026) measure juror error correlation on ordinary preference grading with nothing optimizing against the judges — ρ = 0.944–0.972 for repeated samples of one judge, 0.664–0.706 across a stronger family, and family-mixed juries also below independence predictions — so five jurors buy 0.463 → 0.482 on LLMBar. Condorcet's amplification requires independent voters and LLM judges are not that, optimization or no. The council was never carrying the weight the thesis assigns it; optimization pressure only makes the shortfall adversarial. What is not answered: nothing here tests contested value judgments, where there is no anchor to audit against and hence no way to run this measurement at all.
• The "labs care" dependency is fragile: capabilities can appear or stagnate based on lab priorities you don't control. How should a product hedge against the data-distribution rug-pull?

Sources#

• Andrej Karpathy: From Vibe Coding to Agentic Engineering
• CS329A Self-Improving AI Agents — Part 1: Course Overview — Stanford CS329A lecture 1 (delivered 2025-09-22, published 2026-08-03, practitioner-opinion): the generator–verifier gap naming, human feedback as the bottleneck in unverifiable domains, and verification as the gate on the self-improvement flywheel
• CS329A Self-Improving AI Agents — Part 2: Test-Time Compute Scaling — Stanford CS329A lecture 2 (Azalia Mirhoseini, delivered 2025-09-26, published 2026-08-03, practitioner-opinion): the gap plotted — majority voting plateauing at 10–50 samples against a still-climbing coverage line, reward-model rankers barely denting it, the gap widening from GSM8K to MATH — plus the rarity mechanism (1–3 correct in 10,000), the four-rung verifier taxonomy including output-equivalence checking, and the ~90% manual-correctness caveat on the maths coverage numbers. Figures read off slides in an auto-caption transcript; approximate
• CS329A Self-Improving AI Agents — Part 3: Robust Verification — Stanford CS329A lecture 3 (Azalia Mirhoseini, delivered 2025-09-29, published 2026-08-03, practitioner-opinion): rung 4 opened into its own literature. Used here for the trained verifier's ~400-sample precision decay (and the 100-sample shipped configuration), the PRM's reported reach into the <5%-correct regime, and Weaver as the council-of-judges horizon built with a quality floor and learned weights. Every figure is read off a slide in an auto-caption transcript; the four papers are not in raw/; total COI on Weaver. Machinery on Process vs Outcome Reward Models and Weak-Verifier Ensembling
• CS329A Self-Improving AI Agents — Part 9: Future Research Areas — Stanford CS329A lecture 9 (both instructors, delivered 2025-12-05, published 2026-08-03, practitioner-opinion, auto-caption transcript, ~10.5k words). The closing Q&A on non-verifiable domains: the slow-simulator examples (chip design, wet-lab chemistry, scientific discovery), the instant-verifier requirement and the iteration-count argument behind it, the surrogate-reward-model workaround and its data bound, creativity as the residual truly-subjective case, and Chowdhery's KernelBench gradient from compilable kernels to performance profiling. Practitioner judgment in live Q&A, nothing measured; no paper behind any of it