Every-of-k Reliability

Sources#

• Life After Benchmark Saturation: A Case Study of CORE-Bench
• Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents
• The Agent Said It Was Done. The Database Disagreed.

Summary#

A single benchmark score answers "how does the agent usually do?". Repeating each task k times and counting answers two further questions that pass@1 cannot. pass@k (best-of-k) is the share of tasks solved at least once in k attempts: can it ever do this? pass^k (every-of-k) is the share solved on all k: can you depend on it? For a task with per-attempt success probability p, the three are p, 1 − (1 − p)^k and p^k. Averaged over a benchmark they are three different functionals of the same distribution of p, so two models with the same mean can differ sharply on the other two. A model that is right 60% of the time on every task has pass^20 ≈ 0. A model that is right always on 60% of tasks and never on the rest has pass^20 = 60%. Both score 60% pass@1.

For any agent that writes to real records, the deployment runs once and the operator usually cannot tell which outcome they got (Failures That Look Like Success), so the every-of-k column is the one that describes the deployed system. Best-of-k describes a system with a verifier that can pick the good run, which is a different system.

The largest measurement: ThinkingBox-Bench (October 2026)#

Microsoft's ThinkingBox post (Copilot Studio team with Toloka, Hugging Face blog 2026-10-03, empirical; companion paper arXiv 2608.19741 not ingested) runs 507 synthetic stateful business workflows in five domains (retail, auto insurance, travel, neobank, consulting) 20 times each for 18 models, each run from a freshly reset backend. Grading is on the terminal database state and extracted side effects, by deterministic versioned checks (477 tasks on state alone, 30 adding a narrow binary response rubric). It reports pass@1, pass@20, and observed 20/20, the literal count of tasks that passed all 20 recorded attempts, "no estimator, no smoothing". With n = k the unbiased pass^k estimator used on τ-bench reduces to exactly this count, so observed 20/20 is pass^20 under its usual definition.

Retention: how much of pass@1 survives 20 repeats (Figure 2, chart-read, 13 of the 18 models shown):

Three findings carry the page.

• Breadth and consistency pull apart. Kimi-K3 solves 93.89% of the benchmark at least once (476 of 507, the fewest never-solved tasks in the field) and leads retail pass@1 outright at 82.24%, yet passes only 68 tasks (13.41%) on all 20. Claude Opus 5 solves fewer at least once (79.09%; 106 never solved) and 241 (47.53%) every time. K3 solves 75 more tasks once; Opus 5 solves 173 more dependably. "If you are choosing a model for work that touches real records, pass@20 is the wrong column to look at."
• Near-equal pass@1 hides a 3× reliability gap. GPT-6 Astra (58.3) and Kimi-K3 (57.4) are within a point on pass@1 and 45.6 vs 13.4 on 20/20. GPT-5.4 sits two points below Opus 5.5 on pass@1 and at half its 20/20.
• A newer model can buy accuracy and no dependability. Opus 5.5 beats Opus 5 on pass@1 (67.16 vs 66.50) and on pass@20, and passes exactly the same number of tasks, 241, on all 20 attempts. The post does not say whether they are the same 241 tasks.

Reliability is also domain-shaped. Claude Opus 4.6 scores 68.62% pass@1 on retail and 8.30% on auto insurance, and across the 18 models retail averages 59.52% against auto insurance's 33.83%.

Does consistency track accuracy? Across the field yes, at the frontier only loosely#

Kapoor, Narayanan et al. found outcome consistency rising with accuracy at r ≈ +0.94 across five Codex CLI agents on saturated CORE-Bench (mean pass ≈ 93%), which reads as "reliability comes free with accuracy". ThinkingBox qualifies that. Computed here from the Figure 2 transcription (an inference, not a figure the post reports): across all 13 charted models the pass@1 vs 20/20 correlation is Pearson 0.74 (Spearman 0.84), because the weakest models are unreliable too. Among the seven models at or above 57% pass@1 it falls to Pearson 0.52, and the ordering inverts in places (GPT-6 Astra, sixth on pass@1, is third on 20/20). The two results are compatible once scope is named. Near a ceiling there is little room for p to sit away from 0 or 1, so consistency and accuracy have to move together. On an unsaturated benchmark among frontier models, pass@1 is a weak proxy for the every-of-k number and has to be measured separately.

Pricing consistency: cost per dependable task#

The post prices both quality numbers from each model's recorded token usage at undiscounted OpenRouter list rates (snapshot 2026-09-20, Opus 5.5 at Anthropic's published price). The figures are counterfactual estimates for agent inference only, not bills.

• Cost per successful attempt = cost of 507 attempts ÷ (507 × pass@1). The Pareto frontier is three models: GPT-5.6 Sol ($0.127), GPT-5.4 ($0.131, +3.45pp) and Opus 5.5 ($0.276, +1.80pp). Opus 5 ($0.475 at 66.50%) is dominated by Opus 5.5 ($0.276 at 67.16%). Its 20-run campaign costs $3,206 against Opus 5.5's $1,881, so the newer model is about 41% cheaper per attempt at a higher score.
• Cost per dependable task = cost of the 20-run campaign ÷ tasks passing 20/20. GPT-5.4 is cheapest at $6.80 (128 tasks), GPT-6 Astra $7.45 (231), Opus 5.5 $7.80 (241), GPT-5.6 Sol $9.76 (82), Opus 5 $13.30 (241), Sonnet 4.6 $15.56, GPT-5.2 $19.95, Kimi-K3 $20.68, Qwen3.8-27B $24.36.

GPT-5.6 Sol is the cheapest way to get a right answer and fourth on the cost of a dependable one. "The cheapest way to get a right answer is not the cheapest way to get a dependable one." This is a seventh unit for Cost-per-Task Over Cost-per-Token, and it moves the denominator that Listed-Price Reversal leaves at "per query attempted" two steps further, first to per success and then to per always-succeeding task. One property of the formula is worth stating: per dependable task, a model is charged for all 20 runs of every task it does not pass 20/20, so the cost and the reliability gap compound rather than add.

Every non-agent error is raised to the power k#

pass^k punishes any per-attempt failure source, not only the agent's. This is the condition for reading an every-of-k number, and ThinkingBox is built around it.

• Isolation. Each attempt gets its own MCP session with freshly initialized state, and "two attempts of the same task never share a database row or cached tool state, which is what makes 20-trial comparison meaningful." Shared state between runs would correlate attempts and inflate or deflate k/k in ways that are not the agent's.
• Deterministic judges. A grader with its own run-to-run variance adds false rejects to every attempt. Arithmetic, not a finding from the post: a judge that wrongly fails a correct run 2% of the time caps a perfect agent's pass^20 at 0.98^20 ≈ 67%, and at 5% the cap is about 36%. That is why an every-of-k number from an LLM-judged benchmark needs the judge's own repeat-variance reported beside it, and why ThinkingBox confines its rubric questions to 30 of 507 tasks and narrow binary items.
• Infrastructure errors. The post counts system errors as unsuccessful trials and writes operational failures to an errors sidecar so they can be rerun. Each flaky tool call in the harness is therefore a direct tax on 20/20, and a canonical result has to resolve or account for those attempts.

The post's own closing advice follows from this: "say whether you are reporting best-of-k or every-of-k, and how you computed it."

Other measurements in the corpus#

• A deterministic gate changes the decay rate, not just the level. On τ²-bench airline (Deterministic Pre-Execution Gates, empirical), the vanilla agent falls from pass¹ 29.6% to pass⁵ 8.0% (3.7× decay) and the four-gate suite from 42.0% to 26.0% (1.6×). A pass¹ comparison reads that as +12.4pp of accuracy. The pass^k curve shows a recurring failure mode removed, which is the property ThinkingBox's recommendations (check terminal state before committing, classify tool errors so retries target recoverable ones, cut the tool surface, human approval on irreversible changes) would need to show. The post says it has not measured the lift from any of them.
• A memory intervention reported as Pass^4. IBM's ALTK-Evolve (Read-Time Memory Curation (JitMem), vendor-claim) reports τ²-bench Pass^4 alongside accuracy for a train-mined rule list. At 20 to 40 tasks one task moves Pass^4 by 2.5 to 5 points, which shows how coarse an every-of-k number is on a small task set.
• Saturated-benchmark outcome consistency. The CORE-Bench reliability axis on Measuring Beyond Accuracy Saturation, discussed above.

Limits of the ThinkingBox evidence#

• Vendor-built, but not vendor-graded on its own model. Microsoft's Copilot Studio team built the benchmark and none of the 18 models is Microsoft's own. Microsoft does commercially partner with OpenAI, whose models fill five of the 18 rows. Tasks are synthetic reconstructions modeled on enterprise patterns ("the customers are not real").
• Two different model counts. The prose says eighteen models and Table 2 lists 18. Figure 2 charts 13 and Figure 3 twelve, with the six below 33% pass@1 omitted. Retention and 20/20 values for the omitted models are not published in the post.
• Chart-read values. Figure 2's 20/20 column is read from the chart. The 20/20 counts in Table 3 and the prose (241, 231, 128, 82, 68, …) are exact. Figure 3's pass@20 values are approximate except where the text states them (Kimi-K3 93.89%, Opus 5 79.09%).
• One harness, one temperature policy. All models run in the ThinkingBox orchestrator with a simulated user. Whether 20/20 rank order is a property of the model or of sampling settings and the user simulator is not tested here. The simulated user is itself an LLM, so it is a second per-attempt variance source alongside the agent.

Connections#

• Failures That Look Like Success — why every-of-k is the deployment-relevant column: in production the agent runs once and the silent-failure class gives no signal about which run you got. ThinkingBox supplies that page's largest failure-denominator count (67.24% of 79,853 failed attempts terminated cleanly)
• Measuring Beyond Accuracy Saturation — the reliability axis on a saturated benchmark, where consistency tracks accuracy at r ≈ +0.94. ThinkingBox, unsaturated, finds the link loose among frontier models
• Deterministic Pre-Execution Gates — the cleanest intervention result on pass^k: a deterministic gate cuts the decay from k = 1 to k = 5 from 3.7× to 1.6×
• Cost-per-Task Over Cost-per-Token — cost per dependable task as a further unit, which reorders the per-success cost frontier
• Listed-Price Reversal — that study prices cost per query attempted. ThinkingBox prices per success and per always-succeeding task, the two denominators its residual question asks for, though on list prices and without counting reversals
• LLM-as-a-Judge — a judge's own run-to-run variance is raised to the power k in every-of-k, so LLM-judged repeat metrics need the judge's repeat error reported beside them
• Item Response Theory for LLM Benchmarks — the other "pass@1 is a lossy summary" correction, on a different axis. IRT reads which items a model passes and reorders by ability. Every-of-k reads how close to 1 each item's success probability is. Both re-score a response matrix that already exists, and both reorder near-tied models
• Read-Time Memory Curation (JitMem) — the memory intervention reported in Pass^4, and the granularity problem of every-of-k on 20 to 40 tasks
• Certified Test-Time Scaling Curves — the best-of-k side, made rigorous: a best-of-k curve read to choose a budget needs a simultaneous confidence band. Every-of-k is the column that curve does not describe
• Selection Under a Submission Budget — best-of-k when only n answers may be submitted. Its premise, that a filter can pick the good sample, is the verifier that a single-run deployment does not have
• Kimi (Moonshot AI) — Kimi-K3 is the breadth-without-consistency case
• GPT-6 Astra — the highest retention measured (78%)
• Claude Opus 5.5 — the pass@1 leader that bought no additional 20/20 over its predecessor

Open Questions#

• Is a model's 20/20 rate a property of the model or of the run settings? ThinkingBox runs every model in one orchestrator with one simulated user and the post does not report sampling temperature per model, so the Kimi-K3 vs Opus 5 inversion could partly reflect decoding defaults. Settler: the same 507 × 20 grid rerun with temperature fixed (or swept) per model, or the same models' pass^k on a second stateful benchmark such as τ²-bench.
• Do the production remedies ThinkingBox recommends (terminal-state check before commit, error-classified retries, a smaller tool surface) raise 20/20 by more than they raise pass@1, the way deterministic gates cut pass^k decay on τ²-bench airline? The post says it has not measured any of them. Settler: a ThinkingBox ablation reporting pass@1 and 20/20 for each intervention.

Sources#

• The Agent Said It Was Done. The Database Disagreed. — Tuhin Kundu et al. (Microsoft Copilot Studio team with Toloka; joint Microsoft and Hugging Face post), The Agent Said It Was Done. The Database Disagreed., Hugging Face blog, 2026-10-03, empirical. Cited: Table 1 (metric definitions), Table 2 (pass@1 by domain, 18 models), Figure 2 (pass@1 vs observed 20/20, chart-read, 13 models), Figure 3 annotations and prose (pass@20 for Kimi-K3 and Opus 5; the 75 / 173 contrast), the cost formulas with their GPT-5.4 and GPT-6 Astra worked examples, Figure 4 (cost per successful attempt, Pareto frontier) and Table 3 (cost per dependable task), the "How it works" isolation and judge design, and the errors-sidecar note. Figures are transcribed in the raw's note blocks and were not reopened. The pass@1 vs 20/20 correlations (0.74 / 0.52) and the judge-noise caps (67% / 36%) are computed on this page, not reported by the post. Companion paper arXiv 2608.19741 not ingested, so its Tables 4–5 (standard errors, failure signatures) are known only through the post
• Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents — Reddy, Challaram & Basu, arXiv 2607.07405, empirical: the pass^k table (k = 1…5) and the τ-bench unbiased estimator. Full treatment on Deterministic Pre-Execution Gates
• Life After Benchmark Saturation: A Case Study of CORE-Bench — Nadgir, Kapoor, … Narayanan, arXiv 2606.26158, empirical: the outcome-consistency correlation (≈ +0.94) over five Codex CLI agents. Full treatment on Measuring Beyond Accuracy Saturation