Task Time-Horizon Scaling

Sources#

• Claude Fable 5 and Claude Mythos 5
• CS329A Self-Improving AI Agents — Part 8: Agentic Evaluations and Long-Horizon Tasks
• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
• More compute, more capability: Why AI agent evaluations need to account for test-time compute
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• When AI builds itself

Summary#

The external-benchmark trendline behind Recursive Self-Improvement: the length of task an AI can complete reliably on its own is doubling roughly every four months — accelerated from an earlier ~seven-month doubling. The metric, from METR's time-horizons work, reports the duration over which a model is 50%-reliable at a basket of tasks (the curve looks the same at 80%). It is the quantitative spine of When AI builds itself: where AI Accelerating AI Development shows AI speeding up AI work inside Anthropic, this shows the underlying capability rising on public benchmarks.

The doubling curve#

Model	~Date	Reliable task length
Claude Opus 3	Mar 2024	~4 minutes
Claude Sonnet 3.7	~Mar 2025	~1.5 hours
Claude Opus 4.6	~2026	~12 hours
(projected)	this year	days
(projected)	2027	weeks

Mythos Preview is at the edge of measurability: METR found it could work for "at least" 16 hours and was "at the upper end of what [METR] can measure without new tasks." The trend's acceleration (7-month → 4-month doubling) is the part that matters — it is why the essay argues the loop may close "sooner than most institutions are prepared for."

The June 2026 Mythos-class release pushes further still: Fable 5 / Mythos 5 "can work autonomously for longer than any previous Claude models," and the concrete datapoint is over a week of largely autonomous genomics work (assembling data, designing and training a model, beating a published baseline — see Autonomous Scientific Discovery). A week-long autonomous research run is well past METR's measurable task basket — the metric is now chasing the capability rather than bounding it.

The methodology, and the ancestor curve (as taught, late 2025)#

The "~seven-month doubling" this page treats as the superseded rate is recoverable in full from CS329A lecture 8 (Aakanksha Chowdhery, delivered 2025-11-17, practitioner-opinion, figures read off slides through ASR) — which is also the only source here that describes how the number is built rather than quoting it.

Three task suites, ~170 tasks, chosen to span five orders of magnitude of duration:

Suite	Duration band	~Tasks	Content
SWAA	1–30 seconds	~66	atomic actions (open a file, one small edit)
HCAST	1 minute – 30 hours	~97	diverse software and research-engineering tasks
RE-Bench	up to ~8 hours	~7	full ML-research tasks

The human anchor is the whole instrument: skilled professionals with roughly five years' experience record completion times for successful attempts, and the task's difficulty rating is the geometric mean across baseliners. Chowdhery names the bias this introduces — practitioners fluent in a domain systematically underestimate how hard its tasks are, and their sense of difficulty need not track what a model finds hard. Model runs then supply a success rate per task, a curve is fitted through success-rate-versus-human-time, and the horizon is where that curve crosses the chosen reliability. A student asks the obvious clarifying question and the answer is worth recording: the 50% is per-task success over repeated attempts, averaged across tasks — not a guarantee that every task clears 50%.

The ancestor ladder, at 50% reliability: GPT-2 (2019) ~2 seconds → GPT-4 (2023) ~8 minutes → Claude 3.7 Sonnet (2025) ~59 minutes. (Note the discrepancy with the table above, which records ~1.5 hours for Sonnet 3.7 from Anthropic's own essay. 59 minutes is the figure taught from METR's paper; both are in the corpus and neither is retracted.)

The reliability gap is the lecture's own headline, and it is larger than this page has carried: at 80% reliability the same frontier models sit at roughly 8–15 minutes against 59 minutes at 50% — the lecture gives both "8 to 10 minutes" and "15 minutes" in the same passage, so read it as tens of minutes, not an hour. The 80% curve doubles at a similar rate; it is shifted, not flatter. Chowdhery's gloss: a 50%-reliable agent is "an intern who only completes it half the time", and deploying against real work means living in the 80% column.

Three limitations, named in lecture (all of them METR's own):

• Messy tasks score worse — anything without a single correct answer sits below trend.
• SWE-bench-derived estimates run short. Annotators underestimate SWE-bench tasks, and the models have seen most of the GitHub repositories, so doubling times fitted there are optimistic relative to an unseen repository — contamination showing up as a slope error, not just a level one.
• Models perform like low-context contractors, not maintainers. On internal pull requests, contractors with no prior exposure to a codebase are 5–18× slower than its maintainers — and model performance tracks the contractor times, because the model has never seen that codebase either. The reading the lecture draws is the one worth carrying: the horizon measures what a smart newcomer with no context can do, which is why the context asymmetry and domain expertise keep showing up as the residual human contribution.

The failure taxonomy behind the missing reliability (measured on GPT-4 versus o1, so dated): poor planning (no decomposition into steps), poor tool choice, incorrect mental arithmetic or reasoning, premature abandonment — going round a loop without ever recognising what completion would look like — and outright repetitive loops, where the failed action remains the highest-probability action and gets re-emitted. Reasoning-model training visibly cut the last category (much lower for o1 than GPT-4) and left the others.

And the lecture supplies a countervailing instrument: GDPval, where win rate against a human professional rises roughly linearly over the same two years this page's curve doubles five times. Both were presented in the same session as complementary rather than competing — duration at fixed reliability, versus output quality at fixed task.

The horizon — and its doubling rate — is itself budget-dependent (UK AISI)#

The UK AI Security Institute's July 2026 study (empirical) adds a confound the doubling curve above does not name: a model's estimated time horizon, and the rate at which that horizon doubles, both depend on the compute budget the evaluation allows. A horizon number is only defined relative to a budget.

On AISI's narrow cyber CTF suite, frontier time horizons doubled every 4.7 months since late 2024 when measured at 2.5M tokens/task — close to METR's ~4-month figure on a different (general) basket. But re-fit the same suite at a larger budget and the curve steepens:

• The fitted frontier trend is ~60% steeper at 50M tokens than at 2.5M tokens per task. AISI's own gloss: "the estimated doubling rate is partly a consequence of the compute budget used in the evaluation, not a fixed property of frontier cyber progress."
• At the model level, one recent frontier model's 80% horizon rose from ~40 minutes at 2.5M tokens to ~4 hours at 50M; at the current frontier, raising the budget 2.5M→50M lifts the estimated horizon from ~2 hours to ~14 hours.

The mechanism is a second AISI result: the compute an agent needs scales with how long a task takes a skilled human — a power law with fitted exponent ~0.7–1.0 across AISI's 78 cyber CTFs and METR's 211 software-engineering tasks (a minute-task ≈ thousands of tokens, an hour ≈ millions, a week ≈ billions). It holds even for the cheapest successful run per task, so the floor is set by the work the task requires, not by inefficiency. Because longer tasks demand more compute, a fixed budget runs out on the longest tasks first — so a capped evaluation systematically understates the horizon, and a failure on a long task may mean the run was under-budgeted, not that the model lacked the capability. AISI's cyber range "The Last Ones" (~20 human-hours) went unsolved by every model until the budget reached ≥30M tokens; on Epoch's MirrorCode, a recent model spent up to 1 billion tokens to make progress (weeks of human work) that prior models could not.

This reframes this page's central number. The ~4-month doubling is real, but it is a statistic at an implicit budget; measured at a larger budget the frontier appears to move faster still. Whether the curve is a stable exponential is now entangled with a third question — at what budget? — on top of the exponential-vs-S-curve one below.

METR's own successor: expenditure horizon (July 2026)#

AISI supplied the critique from outside; METR's July 2026 expenditure horizon note (empirical) is the same organisation proposing a replacement instrument, and its "Relation to Time Horizon" section names exactly two limitations of this page's metric as the things it is built to fix:

1. Binary pass/fail throws information away where the task has a continuous score. A continuous score "gives a much more statistically precise measure of capability for each task, i.e. differences between models can be detected with fewer observations."
2. The budget is unspecified. Time horizon "doesn't fully specify a budget or constraints for tokens or other resources… for AI agents or humans" — METR concedes this matters little while agents plateau below human cost and a great deal once they don't, which is the budget-dependence AISI measured above, conceded by the metric's authors.

The replacement runs on one already-heavily-human-optimized problem instead of a basket of human-timed tasks: plot cumulative improvement against cumulative dollars for the agent, plot the local returns to human labour on the same axis, and report the spend at which they cross. On the NanoGPT speedrun humans cost ~$2,500 per 1% speedup; six agent runs at up to $10,000 each re-validate to horizons of $0–$3,300, two of them at exactly zero because their apparent progress was noise.

Three things this does not do, and the differences are worth keeping straight. It is a proof of concept on a single problem — METR says outright that "ideally we would measure expenditure horizons across a few different hard optimization problems," so nothing here replaces the basket. It requires a problem with smooth, continuously-scored returns, which is a narrower class than the tasks this page's basket covers. And it dissolves rather than extends at the interesting end: the metric is only defined while agent returns diminish faster than human returns, so it stops existing at the moment the automated-AI-R&D threshold trips.

Benchmark saturation as the corroborating signal#

The same pattern appears as benchmarks going from near-zero to "saturated" (≈100%, allowing for errors that cap many benchmarks below 100%):

• SWE-bench — hands a model a real open-source codebase + bug report and asks for a change that passes the project's own tests. Low single digits → saturated in two years. (Cf. Claude Opus 4.8: 88.6 on SWE-bench Verified.)
• CORE-Bench — reproduce a published paper's results from its code and data; a prerequisite for conducting original research. ~20% (2024) → saturated in fifteen months.

Saturation is why time-horizon length, not single-benchmark accuracy, has become the more informative capability axis — and why Anthropic retired its task-based AI-R&D benchmarks once models crossed the top human baselines (see AI R&D Autonomy Evaluation (AECI)).

Caveats#

• Infrastructure strain is a leading indicator, not just trivia. GitHub saw ~1B commits in all of 2025; by mid-2026 it saw ~275M/week (~14B/year pace) and is "pushing incredibly hard" on capacity — a downstream signature of the same throughput surge.
• Time-horizon numbers are a 50%-reliability statistic on a basket of tasks; the jaggedness within the basket is real — a model that handles a 12-hour task can still fail a trivial one.
• The generational curve is jagged too. AISI's aggregate "newer models reach further, more reliably, more efficiently" hides a minority regression: on roughly 10–30% of tasks (suite-dependent), a newer model actually does worse than its predecessor (AISI footnote). The reach/reliability/efficiency gains are real on average and jagged underneath — jaggedness across generations, not just within a basket.
• Whether the curve is a true exponential or an S-curve approaching its bend is the explicit uncertainty of Recursive Self-Improvement's first future.
• The release cycle is now shorter than the time to measure the ceiling. Noam Brown (OpenAI, practitioner-opinion): the only way to truly evaluate an agent on a months-long task is to run it that long — but a new model ships every two-to-three months, so a model is retired before anyone has run it long enough to find its ceiling ("nobody actually knows what the ceiling of capabilities are… nobody's run them long enough"). When a long-horizon agent capability shipped, people only realized it mattered a week later, once the first week-long runs finished. The measured horizon lags the true one by structural design — the capability overhang seen from the evaluation side, and the same reason large-scale test-time compute is hard to benchmark to plateau.

Connections#

• Recursive Self-Improvement — this curve, extrapolated, is the quantitative case that the loop could close soon

• AI Accelerating AI Development — the internal-throughput companion to this external-benchmark evidence

• Jagged Intelligence (Ghosts, Not Animals) — the within-basket caveat: long-horizon competence coexists with trivial failures

• The Bitter Lesson — rising capability on general benchmarks is what makes hand-built scaffolding a shrinking advantage

• AI R&D Autonomy Evaluation (AECI) — why saturated task-based benchmarks were retired from RSP determinations

• Build for the Next Model — the forecastable capability curve this measures is what makes "bet on the next release" a rational product strategy rather than a gamble

• Autonomous Scientific Discovery — Mythos 5's week-long autonomous genomics run is a concrete long-horizon datapoint past Mythos Preview's measured 16h ceiling

• Effective Compute Scaling — the compute-side curve this capability-side trendline complements; Whitfill et al. model time-horizon growth under compute projections

• AGI-to-ASI Pathways — a concrete capability trendline feeding the report's quantitative-forecasting and benchmarking-beyond-human agenda

• Intelligence Explosion Dynamics — the metric that makes "is the curve bending toward a singularity, or S-curving?" empirically checkable

• Deep Research Agents — deep research is a long-horizon autonomous task of exactly the kind this metric measures; DRACO grades the report quality at that horizon

• DRACO Benchmark — a sibling capability benchmark (quality of agentic research reports vs. the task length a model sustains); both face benchmark-saturation pressure (DRACO discards >90%-solved tasks)

• Production-Sourced Evaluation — the refresh-from-live-usage method that answers this page's open question of what replaces a saturated task basket

• Repository Exploration Subagent — the SWE-bench family (Multilingual / Pro / Verified / SWE-QA) is FastContext's evaluation surface; the longest-horizon variant (SWE-bench Pro) shows the largest gains, consistent with exploration cost compounding over task horizon

• Planning / Execution Division of Labor — the ceiling (what models can do autonomously, measured here) vs. the realized autonomy users actually grant in practice; Anthropic cites METR's horizons as the rising ceiling its usage data sits below

• Agentic Coding Work-Composition Shift — the rising reliable-task-length ceiling is the upstream cause of usage moving from debugging toward operating/analyzing whole workflows end-to-end

• Conversation-to-Delegation Shift — the usage-side reading of this ceiling: OpenAI's Codex study finds the share of individual users delegating a task estimated at >8 experienced-human-hours rose 2.1%→25.6% since Dec 2025 — task complexity climbing right under METR's measured horizon

• Parallel Agent Orchestration — the long-running-agent runtime margin (p99 OpenAI users ~71 agent-hours/day) is single-agent task duration sitting below this reliable-length ceiling

• Exposure Taxonomy: Observed, Theoretical, Reported, Anticipated — theoretical exposure (what an LLM could do) is bounded above by this reliable-task-length ceiling; the survey's reported/anticipated exposure sit below it

• The Three Loops of AI-Native Building — one anecdotal point on the curve: Andrew Ng's coding agent worked unattended "for around an hour, using a web browser to check what it had built" before returning to him

• Large-Scale Test-Time Compute — the reliable-task-length curve is that thesis measured as a capability; scaffolding models for weeks/months is where the inference budget is spent

• Latent Capability Overhang — the ceiling nobody measures: the release cadence is shorter than the time to run a model to its limit

• Compute-Controlled Benchmarking — reliable task length at a stated budget is a compute-controlled metric; both replace single-number accuracy and both fight benchmark saturation

• Benchmark Score Redundancy — saturation as redundancy: a saturated benchmark has near-zero score spread across models, which is exactly what makes a benchmark trivially predictable in BenchPress's rank-2 matrix — so saturation both erodes this page's metric and makes the score inferable from others

• Measuring Beyond Accuracy Saturation — the "what to do after CORE-Bench saturates" companion: this page cites CORE-Bench as saturating in 15 months; Nadgir et al. take exactly that saturated benchmark and show six non-accuracy axes (construct validity, OOD robustness, efficiency, reliability, model-vs-scaffold contribution, human-agent uplift) still discriminate agents, arguing re-instrument, don't retire

• UK AI Security Institute — the evaluator that measured horizon-and-doubling-rate budget-dependence and the compute-demand–human-time power law; reuses METR's task set

• Noam Brown — source of the "the only way to evaluate a year-long agent is to run it for a year" point

• The Open-Weight Frontier Gap — Arena Elo measures chat preference; whether the 33-Elo open/closed gap holds on long-horizon agentic work is a time-horizon question, not a preference one

• Expenditure Horizon — METR's own continuous, money-denominated successor to this metric, built to fix the two limitations it names here (binary scoring, unspecified budget); measured on one frontier optimization problem rather than a task basket, and undefined past the point where agent returns stop diminishing faster than human ones

• Frontier AI Standards Body — the doubling curve as a regulatory cadence problem: Hassabis proposes a body whose benchmark set defines which models are in scope, refreshed "perhaps quarterly to start." A quarterly refresh is fast for a regulator and slow against this page's trendline, and the proposal does not say what happens to a model that clears the perimeter in the quarter after the benchmark defining it was deprecated

• GDPval Benchmark — the deflationary counterweight taught alongside this metric: win rate against a 10-year professional on real paid work, rising roughly linearly (~12% → ~48%) over the two years this curve doubles repeatedly. Same lecture, opposite extrapolation

• Context Advantage, Not Taste — what the horizon is a horizon of: models track low-context contractor times (5–18× slower than maintainers on internal PRs), so the measured length is a smart newcomer's, not an embedded expert's

• Returns to Expertise in Agentic Coding — the human side of the same contractor/maintainer gap: what the baseliner brings is domain context, and the geometric-mean anchor is built from people who have it

• Reasoning–Acting Interleaving (ReAct) — the compounding shown in miniature on a 2022 benchmark: on WebShop, ReAct's success rate trails its per-step score because "this is a multi-step process, [and] the errors cascade over time" — the same arithmetic this page turns into a horizon

• Governance by Benchmark Threshold: What an Index Must Prove Before an Obligation Can Rest on It — the doubling curve as the reason a benchmark-defined legal perimeter decays structurally rather than unluckily. A ~4-month capability doubling against a ~15-month benchmark saturation time means any perimeter benchmark is obsolete inside a statutory cycle, and the quarterly refresh that would fix it destroys the referential fixity an obligation needs

Open Questions#

• Is the 4-month doubling a stable regime or a local steepening? The trend's shape (exponential vs S-curve) is undetermined. Sharpened (2026-07): AISI adds that the doubling rate itself is budget-dependent — the same cyber suite doubles ~60% faster measured at 50M than at 2.5M tokens/task — so the headline rate is undefined without naming the eval budget. The stability question is now entangled with a budget question, not just an exponential-vs-S-curve one.
• Time horizon is measured on task baskets that themselves saturate; what replaces them once weeks-long tasks become measurable — and who builds those tasks? Partially answered / reframed (2026-07): Nadgir et al. argue don't replace — re-instrument: they take CORE-Bench (cited above as saturating in 15 months) and show it still discriminates agents along six non-accuracy axes after accuracy saturates, so "what replaces a saturated basket" can be "keep it and measure differently" rather than "build a harder one." This addresses accuracy saturation, not the length-metric saturation this page's basket faces, and does not answer who builds the next weeks-long tasks — so it reframes the retire reflex without closing the question. Second partial answer, from METR itself (2026-08): Expenditure Horizon is the metric's own authors proposing a third option — neither "build a harder basket" nor "re-instrument the old one," but change the object measured: drop the basket entirely for a single frontier optimization problem humans are still actively improving, and read capability off where the agent's dollar curve crosses theirs. It answers "who builds the tasks" by not needing tasks to be built — the speedrun leaderboard is the instrument and it refreshes itself. But it buys that with two costs the basket doesn't have: it needs continuously-scored, smooth-returns problems (a narrower class than a general task basket), and it needs a per-problem estimate of the returns to human labour, which on NanoGPT took two contributor interviews, an LLM judge over 82 PRs and a bootstrapped correction factor to produce — and still lands on a number METR calls "highly uncertain."

Sources#

• When AI builds itself — §"Evidence from the outside world" (METR time horizons; SWE-bench / CORE-Bench saturation; GitHub commit-volume footnote)
• Claude Fable 5 and Claude Mythos 5 — "work autonomously for longer than any previous Claude models"; week-long autonomous genomics
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — Noam Brown (No Priors, 2026-06-26), practitioner-opinion: the release cycle is shorter than the time to run a model to its capability ceiling, so the ceiling is never measured
• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT — Cunningham, Shetty, Cheng & Rush (METR, 2026-07-21, empirical): the "Relation to Time Horizon" section, where the metric's own organisation names binary pass/fail scoring and unspecified resource budgets as this methodology's two limitations, and proposes a continuous dollar-denominated crossing point instead. Full treatment on Expenditure Horizon
• CS329A Self-Improving AI Agents — Part 8: Agentic Evaluations and Long-Horizon Tasks — Aakanksha Chowdhery, Stanford CS329A lecture 8, delivered 2025-11-17, published 2026-08-03 (practitioner-opinion, YouTube auto-caption transcript). The teaching account of METR's methodology: the SWAA / HCAST / RE-Bench suite split and ~170-task total, the ~5-years-experience human baseliners and geometric-mean difficulty rating, the GPT-2 → GPT-4 → Claude 3.7 Sonnet ladder at 50% reliability, the 80%-reliability gap, the three stated limitations (messy tasks, SWE-bench underestimation and repo exposure, the 5–18× contractor/maintainer gap models track), and the GPT-4-versus-o1 failure taxonomy. All figures are read off slides through ASR and hedged accordingly; the 59-minute Sonnet 3.7 figure conflicts with the ~1.5 hours in the table above
• More compute, more capability: Why AI agent evaluations need to account for test-time compute — UK AISI (2026-07-02, empirical): the 80% cyber time horizon and its 4.7-month doubling are both budget-dependent (~60% steeper at 50M vs 2.5M tokens; 40min→4hr and 2hr→14hr under budget increases); the compute-demand–human-time power law (exponent ~0.7–1.0) over METR's 211 SWE tasks + AISI's 78 cyber CTFs; "The Last Ones" (~20h) needs ≥30M tokens; MirrorCode's 1B-token run