Discovery-Execution Scaling Forecasts

Sources#

• When Does Longer Reasoning Help? Predicting Mathematical Reasoning Through Discovery and Execution

Summary#

Can short runs tell you what a long run will buy? Hasan, Jain & Samin (arXiv 2610.05322, independent researchers, 2026-10-04, empirical) answer for Olympiad proofs. A short run's success rate alone does not settle it. You also need to measure how much extra compute a model needs to finish a proof once it has the right idea.

They split proof success into two stages. Discovery is the block in which a viable strategy first appears. Execution is how many more blocks the model needs to turn it into a valid proof. Success within K blocks is their discrete convolution:

s(K) = Σ_k p(k) · ε(K − k + 1), with discovery geometric, p(k) = α(1 − α)^(k−1).

Execution is measured directly. Seed a fresh run with a verified strategy sketch of at most 25 words, so discovery is certain at block 1. The curve that results is ε_oracle(K). Discovery is then α ≈ q / ε_oracle(1), where q is the one-block success rate from independent short attempts. Neither measurement looks at the long runs being predicted.

This is a model-based, uncertified forecast checked by paying for the held-out runs. It is exactly the kind of forecast Certified Test-Time Scaling Curves says is the only kind available. Its range is 1 to 8 blocks: an 8× extrapolation, not the 1,000× of Brown's $10 → $10,000 question on Large-Scale Test-Time Compute.

The two propositions, and why they matter beyond this paper#

• Saturated execution reduces to the simple geometric curve. If a discovered strategy always completes within one block (ε(1) = 1), then q = α and s(K) = 1 − (1 − q)^K. This Simple Geometric (SG) curve is the per-problem 1 − (1 − p)^k behind repeated-sampling coverage (Latent Capability Overhang). Under it, every way of spending a fixed budget B on N attempts of K blocks each (NK = B) has the same success probability, 1 − (1 − q)^B. Breadth and depth tie. SG also says continuing an unfinished attempt is worth exactly as much as restarting it.
• Unsaturated execution with rare discovery favors one long run. If ε(1) < ε(ℓ) for some ℓ ≤ B, then as α → 0 the budget is uniquely best spent as K = B, N = 1, to first order in α. Splitting the budget buys no extra discovery chances at first order. A long trajectory leaves more blocks to execute an early discovery.
• Flat but unsaturated execution favors maximum breadth. If ε stays at a constant c < 1, then K = 1, N = B is uniquely optimal (Appendix B.1).

This gives the parallel-versus-sequential argument on Large-Scale Test-Time Compute a mechanism. Breadth beats depth there when the task is hard and the feedback weak. The DE reading: that is the regime where execution is flat, or where discovery is the bottleneck and execution already saturates. Depth should win only where a found idea still needs compute to finish. Wang et al.'s compounding-error explanation for why sequential refinement loses is a separate mechanism, and this paper does not test it.

The setup#

• Problems. 57 non-geometry proof problems. AOBench has 35 from twelve 2026 olympiads and selection tests (China TST 12, Iran TST 5, All-Russian 3, IMO 3, and others), fixed before any model was run. Each has a human-checked reference proof, a sketch of at most 25 words and three explicit load-bearing steps. The other 22 are all the non-geometry problems in IMO-ProofBench Advanced. Geometry is excluded because models turn it into algebra too long for humans to audit.
• Models. Muse Spark 1.2, GPT-5.4, GPT-5.5 and Claude Opus 4.8, all in the same harness. That harness is the Claude Agent SDK 0.1.56 in an isolated Docker container, with local code execution, no web, and compaction at 900k tokens.
• Compute. One block is 200k provider-reported output tokens. The budget is B = 8 blocks (1.6M tokens). A sequential trajectory is one persistent Self-Refine session alternating critique and revision, scored on its last complete proof at each block boundary.
• Measurements versus targets. The measurements are Parallel-8 (24 one-block attempts per problem over three runs, giving q̂) and Oracle-Execution (three 8-block sketch-seeded runs per problem, giving ε̂). The targets are Parallel-1, Parallel-2 and, for two models, Parallel-4. Target outcomes are never used in fitting.
• Grading. GPT-5.6 Sol grades on a near-binary 0/5/6/7 rubric, and 5–7 counts as solved. An IMO-medalist panel re-graded a stratified 50 proofs. They agreed on solved status in 48/50, which is 100% precision and 94.7% recall against the human labels. In every disagreement the automated judge scored lower.
• Predictors. SG uses an unbiased finite-sample pass@NK estimator. R-SG puts a shared Beta prior on q, which is the beta-binomial approach of Kazdan et al. DE is the plug-in convolution. R-DE adds empirical-Bayes priors, a Beta on α and a Dirichlet on the first-completion block. SCT shifts the oracle curve down to the measured fresh success rate.

What measuring execution buys#

The numbers below are aggregate RMSE in solved trials (171 trials per curve; Table 1).

• The GPT models are near execution saturation, so SG is enough. GPT-5.5 completes 162/171 sketch-seeded trajectories within one block and 168/171 within eight. Proposition 1 then predicts geometric scaling, and that is what happens. R-SG is the best predictor for GPT-5.5.
• Opus is the case where execution matters. Its sketch-seeded successes rise from 113/171 to 153/171 over eight blocks. R-SG does no better than SG (7.20 against 7.16), so regularization does not explain R-DE's gain. Combined over N = 1 and N = 2, the RMSE reduction (SG minus R-DE) is 5.31, with a 95% paired bootstrap interval of [2.29, 5.95] (Table 6). That is the only model whose interval excludes zero. Muse's and GPT-5.4's include zero, and GPT-5.5 significantly favors R-SG.
• The contrast also holds inside one model. Split Opus's problems by whether all three sketch-seeded runs finished in block one (Table 2). On the 29 saturated problems, SG beats DE (2.40 against 2.65 at N = 1). On the 28 unsaturated ones, DE cuts error from 5.50 to 1.86 at N = 1 and from 8.25 to 5.22 at N = 2.
• The shape of the oracle curve is not the unaided curve shifted down. SCT, which shifts the oracle curve down to the measured fresh rate, is the worst predictor for every model at N = 1 (average 14.84). The convolution is what does the work. If you flatten Opus's measured execution curve, its RMSE goes from 3.68 to 19.47 (Table 12).
• Problem-level gains are small. Predicting individual trajectories, R-DE moves Opus from 0.93 to 0.86 solved trajectories (Table 5). The forecast is good at the aggregate curve, not at which problem will fall.

Where it breaks#

• Muse at four arms. At N = 2, R-DE predicts Muse's final success almost exactly (77.3 against 77). At N = 4 it predicts 78.5 against 93 observed. A post-hoc check finds the cause in the first block: Muse's one-block success is 27.0% in Parallel-8 runs and 34.8% in Parallel-4 arms. Substituting the Parallel-4 first-block outcomes moves the prediction to 94.4. So the error is not in the execution or discovery model. First-block behaviour does not transfer across allocations. The system prompt tells each attempt its total budget, and the authors suggest this may change Muse's behaviour within the first block. They did not test it.
• Muse also breaks two of the assumptions. A sketch supplied after a 3-block unaided prefix solves 17/71 trajectories, against 34/71 when supplied at the start (−50%). That violates Assumption 1, that execution does not depend on when the strategy arrives. GPT and Opus show 0 to −3.3%. Muse's fitted discovery-decay parameter is γ = 0.692, against 0.972–1.0 for the others, so its late discovery is less likely than geometric. And flat execution improves Muse's DE fit (6.54 to 5.22), so measured execution gains do not carry over evenly across models.
• Which sketch you supply barely matters, but whether it is correct does. An alternate human-checked sketch shifts one-block solves by +13.8% for Muse, +1.8% for GPT-5.5 and −6.8% for Opus. A sketch borrowed from a different problem in the same domain never beats no sketch (Opus: 41 shuffled, 45 none, 73 correct, out of 105). So the oracle condition measures strategy content, not extra prompt text.

Without a sketch for the test problem#

The main forecasts use a sketch written for each problem, which a forecaster would not have for a new problem. In a transfer test, priors are fitted on 38 problems and predictions are made for 19 held-out problems from their short runs alone, over 50 splits. For Opus, R-DE gets a mean RMSE of 5.04 against R-SG's 6.24, and 3.40 when given the test problems' own oracle runs. Muse also improves, GPT-5.4 is roughly tied, and GPT-5.5 is better served by R-SG (Table 14). So a model's execution profile, learned on problems where a sketch exists, partly carries over to problems where none does. The standard deviations come from overlapping splits, not independent replications.

Continue or restart#

Under SG, continuing an unfinished attempt and starting a fresh one are worth the same. DE can tell them apart. Continuing h more blocks adds s(k + h) − s(k), and restarting adds (1 − s(k)) · s(h). R-DE continues when the first exceeds the second. Each choice is scored by regret, the solved trials lost against whichever action did better on that problem. Over checkpoints k = 1…7 with h = 1 (Table 3):

• R-DE's mean next-block regret is 1.50. The retrospectively chosen best fixed action per model scores 1.79, always-restart 2.13, and always-continue 4.30.
• R-DE beats the best fixed action for GPT-5.4, GPT-5.5 and Opus, and loses for Muse (1.57 against always-restart's 0.57). Its largest gains come at 3–4 blocks, where it avoids much of the cost Opus and GPT-5.4 pay for restarting early.
• The paper calls these "descriptive regret comparisons" and puts the uncertainty analysis in a supplement that is not in the raw. Read them as direction, not as an established policy gain.

This is the restart-or-persist decision that Stopping Under a Noisy Verifier makes from verifier and repairer error rates, here made from compute profiles instead.

What this source is and is not evidence for#

• For: short-budget success alone does not determine the value of longer reasoning, at least for one frontier model on hard proofs over an 8× range. Measuring conditional execution adds real predictive information where execution is unsaturated, and nothing where it is saturated. The saturated case is a theorem as well as a measurement.
• Not a cost saving, as run. Per problem, the measurement runs use 24 one-block attempts plus three 8-block sketch-seeded runs, 48 blocks in all. The Parallel-1 target they forecast is three 8-block runs, 24 blocks. The value shown is predicting other allocations and the continue-or-restart choice from one measurement set, not buying a long-run result for less than the long run costs. The no-sketch transfer variant (24 one-block attempts per test problem) costs about as much as the 24 blocks of trajectories it forecasts.
• Not for: forecasting at 100–1,000× budgets, agentic tasks without a clean "strategy" object, or any certified bound. The 57-problem sample and three oracle runs per problem limit precision, and the authors say so. Results vary model by model. Of the four models, only Opus shows an improvement with a confidence interval that excludes zero.
• Provenance. These are three independent researchers with no product COI. They cite their own AutoFyn report for the depth-versus-breadth premise. The AI-use statement says OpenAI Codex helped propose hypotheses, critique the design, formulate the framework, write the harness and draft the manuscript. One structural point: an OpenAI model (GPT-5.6 Sol) grades OpenAI solvers alongside Opus and Muse. The human audit (48/50, with the judge always lower when it disagrees) limits the grading-bias worry but does not split agreement by solver. Release of AOBench, the harness and the fitting code is promised, not done.

Connections#

• Large-Scale Test-Time Compute — the hub whose parallel-versus-sequential findings this page gives a mechanism, and whose cheap-run forecasting residual it partly answers over an 8× range
• Certified Test-Time Scaling Curves — the other half of the forecasting split: that page proves a cheap run cannot certify a high-budget point, and this page is a model-based forecast whose error was checked by paying for the held-out runs
• Latent Capability Overhang — SG is the per-problem 1 − (1 − p)^k coverage arithmetic, and Proposition 1 states when that arithmetic is the whole story. Under unsaturated execution, sequential compute buys something repeated sampling cannot
• Stopping Under a Noisy Verifier — the restart-or-persist decision from the verifier side. This page makes the continue-or-restart choice from measured compute profiles and reaches 1.50 against 1.79 regret for the best fixed action
• Many-Agent Proof Harnesses — the same natural-language olympiad proof setting with a model grader. Here the grader is validated against an IMO-medalist panel (48/50 on solved status, automated scores always the lower ones in disagreements)
• Claude Opus 4.8 — the model whose unsaturated execution (113/171 → 153/171 sketch-seeded over eight blocks) makes measured execution pay off

Open Questions#

• Is Muse's first-block non-transfer (27.0% one-block success in Parallel-8 against 34.8% in Parallel-4 arms) caused by the system prompt stating each attempt's total budget? It would be settled by rerunning Parallel-8 and Parallel-4 with the budget hidden from the prompt and checking whether the gap and the N = 4 forecast error (78.5 against 93) close.
• Does the discovery-execution split predict scaling outside proofs, where the "strategy" is a plan rather than a sketch? For example, agentic coding with a plan-seeded oracle arm. It would be settled by fitting R-DE on a coding or terminal benchmark with plan-seeded runs and comparing held-out RMSE against R-SG, as in Table 1.

Sources#

• When Does Longer Reasoning Help? Predicting Mathematical Reasoning Through Discovery and Execution — Adib Hasan, Lay Jain & Thanic Nur Samin (independent researchers; Samin at IU Bloomington), When Does Longer Reasoning Help? Predicting Mathematical Reasoning Through Discovery and Execution, arXiv 2610.05322, 2026-10-04, 27 pp, empirical. Used: §3 (SG, DE and Propositions 1–2), §3.5–3.6 and Appendix C (estimators, R-DE priors, continue-or-restart rule), §4 (AOBench, harness, 200k-token blocks, measurement vs target runs, GPT-5.6 Sol rubric), Tables 1, 2, 3, 5, 6, 7, 8, 10, 12, 13 and 14, Appendix A.1 (AOBench sources), Appendix B.1 (flat-execution extension), Appendix E.3 (γ fits) and E.7 (Muse N = 4 diagnostics), Appendix F (48/50 human audit) and the AI-use statement. Parse note: PDF-derived (docling 2.126.0 / docling-mlx 0.1.1, confidence_grade: excellent, 15 tables, 8 pictures). Table 2's header is shifted in the raw. The Execution column holds the trial counts, Trials holds SG and the welded SG DE column holds DE, with the N label merged into the row label. Reconciled against pdftotext -layout and the §5.1 prose (5.50 → 1.86, 8.25 → 5.22): the values map correctly once the header is realigned. Tables 1, 3, 6 and 14 have two-level or spaced-digit headers (5. 31 [2. 29, 5. 95]), but their values match the prose wherever both are stated. The Figure 8 confusion matrix is a picture-region table and was not cited. Figure 2 was viewed; it is consistent with the stated oracle counts and with SCT's large positive residuals. The worked example (Appendix G) has OCR-garbled equations and is not cited