Listed-Price Reversal

Sources#

• The price is wrong: AI cost calculation has to consider task completion rates, not just token costs
• The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More

Summary#

A listed-price reversal is a model pair where the model with the lower listed API price ($ per million input tokens plus $ per million output tokens) runs up the higher bill on the same workload. The Price Reversal Phenomenon (Lingjiao Chen, Chi Zhang, Yeye He, Ion Stoica, Matei Zaharia, James Zou; Stanford / Berkeley / CMU / Microsoft Research; arXiv 2603.23971 v2, 2026-05-28, empirical) is the corpus's first systematic count of how often it happens:

• Grid. 8 reasoning models: GPT-5.4, GPT-5.4 Mini, Gemini 3.1 Pro, Gemini 3 Flash, Claude Opus 4.7, Claude Haiku 4.5, Kimi K2.6 and MiniMax-M2.7. 12 benchmarks: 9 single-turn (AIME, ARC-AGI, ArenaHard, GPQA, HLE, LiveCodeBench, LiveMathBench, MMLU-Pro, SimpleQA) and 3 multi-turn agentic (Terminal-Bench 2.0 on Terminus 2, Cybench and GAIA on ReAct). Pricing is a single snapshot from 2026-05-01. All calls ran February to May 2026, more than $5,000 in all.
• Rate. 28 pairs × 12 tasks = 336 comparisons, and 106 (32%) reverse. The rate runs from 11% on ArenaHard to 57% on MMLU-Pro. On the agentic tasks it is 25% (Terminal-Bench 2.0), 35.7% (Cybench) and 28.6% (GAIA), per Table 6.
• Size. The abstract says reversal magnitude reaches 28×. The body's worked cases are smaller. Gemini 3 Flash lists 1.7× cheaper than Claude Haiku 4.5 and costs 4× more on MMLU-Pro ($219 vs $57). Kimi K2.6 lists 72% cheaper than GPT-5.4 and costs about 2× more on Cybench ($61 vs $34).
• Headline. Summed over all 12 tasks, Gemini 3 Flash ($3.50 listed) costs $705 against GPT-5.4's $509 ($17.50 listed). It is 80% cheaper on the price list and 38% dearer on the bill.

The authors' conclusion is that listed price is an unreliable proxy for cost, and that the realized cost distribution should be reported as an evaluation axis alongside accuracy. That is Compute-Controlled Benchmarking's demand, now with a count of how wrong the price-only shortcut is.

Why the bill inverts: thinking tokens and turns#

The paper attributes each pair's cost gap ΔC across eight factors: turns T, average fresh-input, output, cached and thinking tokens per turn, and the three prices. It uses a Shapley value, which it shows is the unique attribution satisfying four fairness properties. Because cost is multilinear in the factors, the attribution has a closed form.

• Single-turn tasks: thinking tokens. In Figure 3, more than 95% of the cost gap among reversed single-turn pairs is attributed to thinking tokens (the "thinking tax"). Table 5 shows the size of it. On MMLU-Pro, Gemini 3 Flash spent 71.5M thinking tokens against GPT-5.4's 1.8M and Opus 4.7's 1.8M. On LiveCodeBench it spent 53.3M against GPT-5.4's 3.3M. The per-query example is a single MMLU-Pro problem where Gemini 3 Flash used over 60,000 thinking tokens and GPT-5.4 used 25.
• Agentic tasks: turns and replayed context. On Cybench, turn count accounts for more than 80% of the Kimi K2.6-vs-GPT-5.4 reversal. To find the same security issue, GPT-5.4 Mini took 7 turns and Kimi K2.6 took 57. Appendix E names the mechanism. On all three agentic benchmarks the largest cost component is the replayed history (input tier), so per-task cost grows super-linearly with turns. Cheaper-listed models hit the 1,000-message limit far more often. This is the same structure Cache-Aware Agent-Loop Cost (TurnCost) measures on 600 coding trajectories (turn count carries 84% of log-cost variance, exponent 1.23), here reached by attribution rather than regression and across vendors.
• Difficulty amplifies it. The paper buckets queries by how many of the 8 models solve them. On easy queries (6–8 solve), the gap is small. On hard ones (0–2 solve), turns dominate the multi-turn gap. Appendix E adds that reversal gaps on incorrect runs are 3–10×, against within 2× on correct ones. The expensive half of a reversal is mostly paid for failures.

The over-deliberating cheap model is the mirror of the over-deliberating strong model in Scale-Dependent Prompt Sensitivity. Here, cheapness per token is spent back on thinking volume.

Within-vendor tiers: the start-smart claim, checked on a third party's grid#

Cost-per-Task Over Cost-per-Token carries Anthropic's claim that a stronger model is often cheaper per task. The paper's Figure 6 prints per-benchmark dollar totals for every model. Reading the labels for the three same-vendor tier pairs (vault arithmetic on chart labels, rounded to whole dollars):

So the inversion within a vendor's own line happens in 8 of 36 tier comparisons (22%). Three of the eight are agentic, out of 9 agentic tier comparisons. Summed over all tasks, the cheaper tier is cheaper in all three families. The start-smart rule is therefore a minority outcome on this grid. It is not a rule. Its frequency is highest on the long-horizon tasks where its mechanism (fewer turns) is supposed to operate, and that matches Databricks' Opus-over-Sonnet result on real coding tasks.

Cost is a distribution#

Repeated rollouts of one (model, query) pair span up to 9.7× between the cheapest and dearest run, with within-query coefficient of variation routinely above 0.3 (Figure 5, AIME and Terminal-Bench 2.0). Any single reversal verdict can flip across runs.

The paper recomputes the reversal rate at each quantile of the per-query cost distribution (Table 6, Figure 9). Pooled, it is 31.5% at the mean, 39–40% across Q15–Q40 (the range a per-query budget actually meets), and lowest at Q99 (25.0%). On heavy-tailed workloads the mean behaves like a high quantile, so it under-reports the reversal a typical request sees. The shape differs by workload. Reasoning benchmarks give an inverted U. Short-answer and chat benchmarks stay flat near 30%. Agentic benchmarks peak in Q40–Q85. The authors propose predicting the per-query cost distribution as an open problem, and quantile-level attribution as a second one with no closed form yet.

What the study does not measure#

• No quality adjustment. Cost is per query attempted, right or wrong (Limitations iii). The authors argue that adding quality "would only strengthen" the findings because stronger models usually answer better. That is asserted, not measured. It holds only for reversed pairs where the pricier model is also the more accurate one, and the paper does not report accuracy per pair. A cost-of-pass re-scoring, the metric of the authors' own group (Erol et al., ICLR 2026), is not in the paper.
• Benchmarks, not production. Public task sets on off-the-shelf scaffolds, one reasoning setting per model (max or high effort, Table 4), one price snapshot. The authors state that rates will move with re-pricing and leave the effort/cost trade-off as future work.
• Listed price is p_in + p_out. That sum is a convention the authors attribute to practitioner practice. It weights input and output equally whatever the workload's mix.

Version history matters#

arXiv 2603.23971 v1 (2026-03-25) and v2 (2026-05-28) are different studies under one ID. v1 reported 9 single-turn tasks, 21.8% reversal, and Gemini 3 Flash 78% cheaper listed than GPT-5.2 and 22% dearer in practice. It also reported that removing thinking-token cost cut reversals by 70% and raised Kendall's τ between price and cost rankings from 0.563 to 0.873. v2 swapped the roster (GPT-5.4, Opus 4.7, Kimi K2.6, a May price snapshot), added the three agentic tasks and the Shapley framework, and reports 32%.

The rise from 21.8% to 32% is not the agentic tasks' doing. v2's nine single-turn datasets alone average 32.1% (Table 6, vault arithmetic), so the new roster drove it. Two v1 remnants survive in v2's text: §3 still says "roughly one in five cost judgments … would be wrong" (v1's 21.8%) beside its own 32%, and the 28× magnitude appears in both abstracts but is never derived in v2's body. The Register's July report dates the finding to "March" (v1) while quoting v2's numbers.

Internal inconsistencies worth knowing before quoting#

• Figure 2a's per-benchmark rates agree with Table 6's MEAN column except Cybench (39% vs 35.7%) and GAIA (32% vs 28.6%).
• Figure 1's 12-task totals reproduce from Figure 6's per-benchmark labels for Gemini 3 Flash (705), Gemini 3.1 Pro (678 vs 677) and GPT-5.4 (511 vs 509). They do not reproduce for Haiku 4.5 (labels sum to ~508, Figure 1 says $421), GPT-5.4 Mini (~188 vs $325), Kimi K2.6 (~453 vs $422), MiniMax-M2.7 (~124 vs $148) or Opus 4.7 (~720 vs $708). The gaps exceed whole-dollar rounding. The headline Gemini 3 Flash vs GPT-5.4 example is one of the pairs that does reproduce, so cite it. Treat the other models' Figure 1 totals as unverified.
• Figure 2's caption calls the model "Gemini 3.1 Flash". The paper's roster has Gemini 3 Flash.

Connections#

• Cost-per-Task Over Cost-per-Token — the thesis this paper generalizes across vendors. It also settles that page's residual question for benchmarks: the pricier model costs less in about a third of pairs, and in 22% of same-vendor tier pairs
• Compute-Controlled Benchmarking — the falsification test that page's price-row question proposed, run on a superset of the models involved. Price-implied cost rankings break in 25–36% of pairs on agentic benchmarks. The paper's call to report cost distributions alongside accuracy is that page's demand
• The Price of Fixed Capability — Epoch's reason for measuring cost per question rather than price per token (reasoning models get their scores by spending tokens) is this paper's mechanism, counted. The 9.7× within-query spread is a noise floor under any per-question cost series
• Cache-Aware Agent-Loop Cost (TurnCost) — the same agentic mechanism (turn count and replayed history, super-linear in turns) measured on a different corpus. TurnCost regresses cost on turns within open-weight scaffolds, and this paper attributes the gap between proprietary model pairs
• Scale-Dependent Prompt Sensitivity — the same overthinking failure from the opposite direction. There, large models over-deliberate on easy items. Here, cheap-per-token models over-deliberate enough to erase their price advantage
• Kimi (Moonshot AI) — Kimi K2.6 is the paper's sharpest agentic reversal (57 turns vs GPT-5.4 Mini's 7 on one Cybench task, the most expensive model on Cybench at the third-cheapest listed price)
• Claude Opus 4.7 — the most expensive model listed and in total, and cheaper than Haiku 4.5 on 4 of 12 benchmarks including Terminal-Bench 2.0

Open Questions#

• Can a per-(model, query) cost-distribution predictor beat the within-query noise floor, with calibrated quantiles? The paper poses it and offers no baseline. Settler: a published predictor evaluated on held-out queries, for example on this paper's released per-trial dataset, reporting quantile calibration.
• Does the ~32% reversal rate survive when each model runs at its cheapest adequate reasoning effort rather than max or high (Limitations ii)? Effort is the dial Cost-per-Task Over Cost-per-Token treats as a second axis. Settler: the same 12-task grid re-run at two or more effort levels per model.

Sources#

• The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More — Chen, Zhang, He, Stoica, Zaharia & Zou, The Price Reversal Phenomenon: When Cheaper Reasoning Models End Up Costing More, arXiv 2603.23971 v2 (2026-05-28; the raw's published: was corrected from v1's 2026-03-25 to this date on 2026-10-21), 22pp, empirical. Cited: abstract, §2–5, Appendix A (Tables 2–4), B (Table 5, Figure 6), D (Table 6, Figure 9), E, G; Figures 1, 2, 3, 5 and 6 viewed (Figure 6's bottom row re-read from a 300-dpi render of PDF p.16). v1 figures come from the arXiv v1 abstract page, fetched at compile. Parse warning: Table 2 (datasets) is collapsed and welded in the raw: the Category and Description cells of the nine single-turn rows are merged into the AIME row, and Terminal-Bench/Cybench task counts (89/39) are welded into one cell. Table 4 (decoding configuration) is collapsed into one row. Both were reconciled against pdftotext -layout and match the PDF in order. Tables 3, 5 and 6 parsed clean and were spot-checked against the PDF. The within-vendor table above is vault arithmetic on Figure 6's printed labels, not a figure the paper prints. COI: none commercial in the models compared. Zaharia is Databricks' CTO and Stoica a Databricks co-founder. Databricks benchmarks the same question on its own codebase (The price is wrong: AI cost calculation has to consider task completion rates, not just token costs) and sells a harness-swapping product (Omnigent), though no Databricks model or harness is in this grid
• The price is wrong: AI cost calculation has to consider task completion rates, not just token costs — The Register, 2026-07-13 (case-study, secondary): relays "about a third" and the Gemini 3 Flash 80%/38% example. Both match v2. Its "back in March" is v1's date, and v1's numbers were 21.8% and 78%/22%