Latent Capability Overhang

Sources#

• More compute, more capability: Why AI agent evaluations need to account for test-time compute
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown

Summary#

If capability scales with inference budget (Large-Scale Test-Time Compute) but nobody spends a large budget, then the models already released can do far more than anyone has demonstrated. Noam Brown calls this out directly: "nobody had explored sufficiently what happens if I put $100,000 worth of compute into [a released model]." The capability is latent not because it's absent but because extracting it costs money and patience that almost no one spends (practitioner-opinion).

The worked example: the Erdős unit distance conjecture#

OpenAI disproved the Erdős unit distance conjecture using an internal model — per Brown, "a pretty big deal in the math community," the first such problem a lot of mathematicians had spent serious time on, and solved "at a budget that was dirt cheap" (they trained a new model, were curious, and ran it at low budget). The revealing part is what came after: once the result was announced, people found you could get the same disproof out of the already-public GPT-5.5 — not by asking directly, but by scaffolding it (ask it to list attack strategies, tell it to explore the promising one, iterate). Brown estimates a general-purpose scaffold that arrives at the disproof would cost "a thousand to $10,000 to $100,000" — expensive, but possible, and possible before OpenAI did it. The capability was sitting in a released model; nobody had paid to reach it.

This is the informal-but-checkable sibling of DeepMind's Lean-verified Erdős work: DeepMind resolved formalized Erdős problems with a compiler certifying every step, while OpenAI's disproof was produced by a general model under human steering and verified after the fact. Both are 2026 evidence that frontier models now contribute to open mathematics — one via a sound verifier, one via test-time-compute search.

Measured, not just anecdotal (UK AISI)#

The Erdős story is practitioner-opinion — a single reconstructed anecdote. The UK AI Security Institute's July 2026 study (empirical) supplies the measured overhang on released models, and it is exactly the shape this page predicts: capability that a standard fixed-budget evaluation never reveals because nobody spent enough.

• ~8% of AISI's narrow cyber tasks were solved only once the per-task budget reached ≥10M tokens (some up to 50M). "At smaller budgets, those successes would have been invisible." The latest models kept climbing at 100M+.
• "The Last Ones" (~20 human-hours) went unsolved by every tested model until the budget reached ≥30M tokens — a whole capability class hidden below that line.
• On public benchmarks the same shape holds: 1M→10M tokens buys +~25% on software engineering and +~22% on maths/academic tasks.

This is the overhang quantified rather than asserted, and by a government third party independent of the OpenAI-sourced framing: for released models, a routine eval budget leaves a measurable slice of real capability latent, exactly because extraction costs tokens nobody spends.

The release-cycle interaction: why nobody bothers#

The overhang persists because of a rational disincentive. The cost of any given capability drops 10–100× with each model release cycle (every two-to-three months), so spending $100K to extract something today is often dominated by waiting for the next model to do it for a fraction. This is the "go on vacation, come back two months later, and it's a thousand times cheaper" meme — the pessimistic twin of building for the next model. Brown half-endorses it: OpenAI is "in a period where progress is very fast."

But OpenAI's institutional choice runs the other way, and for a reason: it actively discourages its mathematicians and physicists from spending all their time pushing current models to their limits on open problems. The stated logic is opportunity cost at the frontier — "the focus should be on how do we make even more capable models… so that all the scientists in the world can use these models to solve the problems themselves." Mining the overhang is a distraction from widening it.

The evaluation blind spot it creates#

The overhang is also why nobody knows the ceiling of the current models. Pushing a model to its limits takes two-to-three months; a new model ships every two-to-three months; so each model is retired before anyone has run it long enough to find out what it could do. Brown's example: when a long-horizon agent capability shipped, people didn't realize it was a big deal until runs that took over a week finally finished — a week after release. The measurement lag is structural, and it compounds the safety-evaluation gap: if you can't afford to find a model's capability ceiling before the next one lands, you also can't find its dangerous-capability ceiling.

Connections#

• Large-Scale Test-Time Compute — the root cause: the overhang exists only because capability scales with a budget nobody spends
• Compute-Controlled Benchmarking — the reporting twin: grids under-report capability because they under-spend, the same axis this page reads as latent upside
• AI-Driven Formal Proof Search — DeepMind's Lean-verified Erdős results; the formally-certified sibling of OpenAI's informal unit-distance disproof
• Build for the Next Model — the product-strategy inverse: "wait for the next model" (mine later, cheaper) vs. "build for the next model" (prototype now, let the release close the gap)
• Task Time-Horizon Scaling — the ceiling nobody can measure: the release cadence is shorter than the time to push a model to its limit
• Responsible Scaling Policy Evaluations — the safety cost of the blind spot: an unmeasured capability ceiling is also an unmeasured dangerous-capability ceiling
• Open-Weight Elicitation Irreversibility — the overhang with no recall mechanism: for published weights, the elicitation budget is unbounded and permanent
• Inference Efficiency as Capability — the 10–100× per-generation cost drop, disaggregated into the levers that produce it
• UK AI Security Institute — the government evaluator that measured the overhang: ~8% of cyber tasks solved only at ≥10M tokens, "The Last Ones" only at ≥30M
• Noam Brown — the source
• OpenAI — the lab that disproved the conjecture and that chooses not to mine the overhang

Open questions#

• If cost falls 10–100× per release, when is it ever rational to spend big extracting a capability now rather than waiting? (For a lab racing a competitor to a specific result, "now"; for everyone else, rarely — which is why overhangs accumulate.)
• How large is the overhang in a given released model — is there a way to estimate the ceiling without paying to reach it? (This is the projection question of Large-Scale Test-Time Compute read as a safety instrument.) Sharpened (2026-07): AISI is actively working both halves — forecasting high-budget performance from cheaper runs, and defining "minimum informative budgets" (a budget declared sufficient only once reach stops rising with more compute, which is precisely the "have we reached the ceiling?" test). Unsolved, but now an active government research program rather than an open wish.
• Who audits released models for latent dangerous capability, given the same disincentive discourages spending the budget to find it?

Sources#

• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — No Priors interview (2026-06-26); the Erdős unit distance disproof, the "$100K into a released model" observation, the cost-drops-per-cycle meme, and OpenAI's choice not to mine the overhang (practitioner-opinion)
• More compute, more capability: Why AI agent evaluations need to account for test-time compute — UK AISI (2026-07-02, empirical): the measured overhang on released models — ~8% of cyber tasks solved only at ≥10M tokens (some 50M), "The Last Ones" only at ≥30M, +25%/+22% from 1M→10M tokens