Harness Swap vs. Rerun Noise (What a Harness Buys)

Sources#

• What Does a Harness Buy? Tokens, Mostly

Summary#

Yangze Liu & Zhongyi Han (Shandong University), What Does a Harness Buy? Tokens, Mostly, arXiv 2610.04433, 2026-10-03, 22pp, empirical (What Does a Harness Buy? Tokens, Mostly). The question is the one every harness claim rests on: does swapping the harness under a fixed model move a score more than running the same harness again? Earlier harness comparisons, including Harness Tax: Coding-Agent Cost Multiplies Across Harnesses While Success Barely Moves, ran each cell once or twice. This paper runs rerun pairs of identical configurations and reads every harness gap against them.

Four findings:

1. Pass rate: a harness swap is a rerun. On the 45 hardest tasks, swapping the harness flips 13% of task outcomes at the median, exactly what rerunning one harness flips. The tasks a harness wins in one run are not the ones it wins in the next. On a 447-task pool, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within ±5 points on both models run there.
2. Every harness effect that clears the noise is a loss. OpenCode trails by up to 9.2 points on the pool, and about half of that gap is trials its output cap cut off with no recovery. No harness was measured winning a task.
3. The harness sets the bill. With the same model, tasks and price list, cost per task differs by up to 3×. The gap is set at the first call by a fixed preamble of system prompt and tool schemas, and multiplied by the step count. Per-call tool output barely differs.
4. The resolution a harness comparison needs. At the observed 14% discordance, 45 tasks catch a 12.9-point gap only half the time and no gap at 80% power. 447 tasks resolve 5.2 points, which is still coarser than many advertised harness gains.

Design#

• Harnesses, as shipped and version-pinned. Claude Code 2.1 (2.1.259–2.1.273), mini-SWE-agent 2.4.6 (one bash tool, no context management, hard stop at the window), OpenCode 1.18 (typed file/search/edit tools, pruning plus auto-compaction). No harness sends sampling parameters. One budget is held fixed: 300 model calls per trial.
• Models. Qwen3.6-35B-A3B and Qwen3.8-27B served locally (vLLM, bf16), plus DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview through vendor APIs. Claude Opus 5 runs inside Claude Code only, as a top anchor. DeepSeek's own CLI harness (dsh) runs on its own model as a home-field check.
• Tasks. SWE-bench Verified minus 8 tasks whose verifier needs a web service (492). hard45 is every task in the two hardest difficulty bands (42 + 3), taken whole. pool447 is the rest, run on the two locally served Qwen models only.
• Offline. The agent phase reaches only the model endpoint, enforced at the network layer (see Leakage below).
• Scoring. A cell's score is its first scheduled run. Task outcomes are paired across harnesses on the same model: McNemar's exact test, Newcombe intervals, two one-sided equivalence tests at δ = 5 on the pool. Reruns are kept apart and used only to measure how far a score moves on its own. Cost comes from each trial's own token counts (cache-hit input, cache-miss input, output) at the vendor's list price.

Pass rate: the harness gap is the size of the rerun gap#

pool447 (Table 1, reconciled against the PDF; see the parse note under Sources):

The CC − mini 90% intervals are [−4.7, +1.1] and [−3.7, +1.0], both inside ±5, so Claude Code is at most about one point ahead. Under Holm's correction over the six pool comparisons, the two Qwen3.8-27B OpenCode gaps survive and the Qwen3.6 one does not.

hard45 (Table 2, first runs, passed/45): Qwen3.6 11 / 15 / 11 (CC / mini / OC); Qwen3.8 19/43 / 17 / 17; DeepSeek-V4-Flash 21 / 22 / 19; GLM-5.3-Flash 22 / 20 / 18; HY4-Preview 23 / 26 / 21; Opus 5 in Claude Code 36. Across harnesses the spread is two to five tasks for every model, and rerunning a cell moves it by up to three. mini-SWE-agent leads on three models and Claude Code on two. Every paired interval on this set covers zero, including HY4-Preview's −6.7 (CC − mini) and +11.1 (mini − OC), p =.45 and.12.

Flip rates put reruns, harness swaps and model swaps on one scale (Figure 2, viewed). The flip rate of a pair of runs is the share of tasks whose outcome differs between them:

On pool447 the one rerun pair flips 13% and the six harness pairs 9–16%.

The flips do not replicate. If a harness owned a set of tasks, a split in run 1 would repeat in run 2. Across the six harness pairs run twice on both sides, the 12 tasks split in both runs went 5 the same way and 7 the opposite way (binomial p =.77; a task-level sign-flip permutation gives p =.821). A real effect leaves a different trace: for a model swap under the same harness, 10 of 15 such tasks split the same way both times, even though Qwen3.8-27B leads Qwen3.6-35B-A3B by only two to ten tasks. On hard45, the harness gap behaves like a fresh draw of the same noise.

This is the run-to-run term that SWE-Bench Pro V2's variance split leaves as 65% of outcome variance, measured here as a flip rate a reader can compare against a harness claim directly.

The resolution a harness comparison needs#

Power is computed from the pooled 14% discordance between harness pairs (Figure A2):

• 45 tasks: a 12.9-point gap is caught half the time. No gap reaches 80% power, because fewer than six discordant tasks split one way cannot reject at all. Below 55 tasks nothing reaches 80% power at this discordance, and at 45 the ceiling is 62%.
• 447 tasks: 5.2 points detectable at 80% power.
• The asymptotic sample-size formula is optimistic by half a point to a point everywhere.

So a harness gain of a few points read off one run of a few dozen tasks is inside the noise, and some published gains sit below even the 447-task resolution. The paper's examples are a vendor's +2.4-point gain over Claude Code (Zhipu AI's ZCode) and a 0–8-point three-harness study on 50 tasks, one run per cell (Vats & Golev, not in this corpus). The same arithmetic applies to the harness-evolution gains on Agent-Authored Harness Optimization, most of which are reported without a rerun. The cheapest control is a second run of the same cell, the one most comparisons skip, and the authors recommend reporting it beside any harness gap. Every-of-k Reliability makes the related point from the deployment side: a single run's pass rate hides per-task variance.

Every harness effect measured is a way to lose a task#

• An output cap with no recovery (OpenCode). OpenCode caps one response at 32k output tokens. When the model's reasoning runs past it mid-step, OpenCode does not re-prompt. On pool447 with Qwen3.8-27B, 38 of 447 trials contain such a cut, 34 of them on the final step. Those pass at 7.9% against 75.9% for the rest, and 32 of the 34 end with the repository untouched. Claude Code hits the same cap on the same model in 52 trials (66 events). 64 events are followed by its own resume prompt, and one trial ends on the cap. Scoring only uncut tasks roughly halves OpenCode's gap (per-protocol Δ +3.4 and +4.6, against +7.9 and +9.2).
• The recovery transplant is null on its sample. Patching OpenCode's wrapper to re-invoke with Claude Code's resume string, verbatim, on the 41 affected pool tasks (two attempts each, stock vs patched): mean per-task reward 0.585 → 0.634, six tasks up, two down, p =.29. The prompt fired on only 15% of patched trials, against 33 of 41 parent trials that had been cut, so the runaway reasoning belongs to the run rather than the task. Recovery cost 1.23× the steps and 1.30× the input tokens.
• Stopping sooner (OpenCode). On Qwen3.6-35B-A3B over the pool, where no trial is cut off, OpenCode ends after a median of 28 model calls against 71 (mini) and 55 (Claude Code), and still trails mini by +4.7. It also stops soonest on GLM-5.3-Flash. The paper says the data do not identify what makes the model stop.
• No context management (mini-SWE-agent). On Qwen3.8-27B, mini runs into the window on 11 of the 45 hard tasks and 18 of the pool. Most are symptoms: only 3 and 6 of those were passed by either other harness, so recovering all of them would move mini by 6.7 and 1.3 points. On rerun, 7 of the 11 overflow again.
• Limits of the full harnesses. Claude Code and OpenCode compact and lose at most one trial to the window. Claude Code loses at most two per arm at the step cap. OpenCode hits the wall clock on six hard tasks on Qwen3.8-27B, but it had a 5,760 s limit against Claude Code's 36,000 s there, and all six were still making tool calls with a patch on disk. The paper reports this as an uncontrolled factor, and dropping those tasks does not shrink the gap.
• Offline search. Shipped OpenCode downloads ripgrep at first use, so its grep/glob tools fail offline. The reported arm pre-places the binary. The shipped arm is 11 points worse on Qwen3.6 hard45 and higher on the other Qwen model.

The design reading: harness engineering pays where the losses are, in a recovery step at each hard limit. The parts that get the most attention (prompt, tool surface, compaction policy) differ most across the three harnesses, and none of those differences bought a score gain. This matches the exception on SWE-Bench Pro (Lineage and the V2 Reissue)'s harness study, where the one scaffold effect clearing significance was also a loss (Inkling trailing under Pi by 5.8–7.5 points).

The harness sets the bill#

A bill is per-step input times steps, and the harness moves both terms (Figure A3; Table A4, GLM-5.3-Flash on hard45, 45 runs per harness):

• What the tools return is not the difference. Per call, the three harnesses hand the model about the same amount of text, at the mean and at p90. What differs is the harness's own text, paid again on every step. The preamble is a residual (first-call prompt minus the 554-token issue statement), because the harnesses' system prompts are not in the trajectories.
• Claude Code also re-sends the model's reasoning. Its run accumulates 58,359 tokens of model text and reasoning against mini's 1,715. The per-step reconstruction closes for Claude Code only if reasoning blocks are counted as re-sent history. That term is inferred from the fit, not observed. Reconstructed input lands within +6%, −13% and −6% of measured in the three arms.
• Lines start apart and climb at similar rates (Figure 4, viewed). Per-step growth is 937 / 648 / 681 tokens on GLM-5.3-Flash and 1,308 / 1,037 / 1,116 on DeepSeek-V4-Flash (CC / mini / OC), within a factor of 1.5. Of Claude Code's slope, 82 and 187 tokens per step are kept reasoning. So the bill is set at the first call and multiplied by the step count. Claude Code compacts in a fifth to a third of its trials. mini never compacts and overflows instead.
• Which term wins depends on the model. On GLM-5.3-Flash, Claude Code and mini take similar step counts, so per-step weight decides and Claude Code costs ~1.9× mini. On DeepSeek-V4-Flash, mini takes 141 steps to Claude Code's 104, the lines run nearly parallel, and the lightest harness moves the most tokens: 0.98 CNY per trial under Claude Code against 1.02 under mini.
• The cache price scales the bill and does not reorder it. 96–99% of input tokens are cache hits in every arm on both vendor models. Zhipu charges 0.287× the miss price for a hit, so cached input dominates the GLM bill. DeepSeek charges 0.033×, so the cached transcript is under half the bill and output is as large a term. Repricing each model's tokens under the other vendor's list keeps the harness order: on the GLM tokens Claude Code is 1.86× mini at Zhipu's prices and 2.09× at DeepSeek's; on the DeepSeek tokens 0.95× and 0.79×. What flips Claude Code from cheaper than mini to twice its cost is the model's step profile, not the price list.

The numbers the authors say to carry between vendors are three: the first call, the growth per step, and the step count.

What was looked for and not found#

• No home-field advantage. DeepSeek's own harness on DeepSeek-V4-Flash passes 21 of 45, the same as Claude Code and one fewer than mini, at 1.04 CNY per trial, the top of the range. This agrees with Harness Tax: Coding-Agent Cost Multiplies Across Harnesses While Success Barely Moves's finding that a provider's own harness is not reliably a model's best pairing.
• Claude Code lengthens the transcript without deeper reasoning. On Qwen3.8-27B over the pool, reasoning is 82.5% of output under Claude Code and 82.9% under OpenCode, while total output per trial is 1.23× under Claude Code.
• Tool ablations and a year-old release move only the bill (Opus 5 in Claude Code, hard45). Bash-only (file-edit tools removed, two runs) and no-shell both pass within one task of the three control runs' 35–36. The within-control flip floor is 6/135. Release 1.0.100 (2025-09-01) passes 34 at 0.44× the cost, because it did not enable extended thinking by default. Its median output is 0.24× the controls' and its reasoning is zero. Costs are Claude Code's self-reported list-price figures on a subscription, so they are an effort index, not spend.
• The one interaction: thinking. With thinking switched off at the serving layer on Qwen3.6-35B-A3B (two runs per side), Claude Code and OpenCode move +2.2 and +3.4 points with as many tasks up as down, while mini-SWE-agent loses 11.1 points (10 down, 2 up, p =.04). On HY4-Preview inside Claude Code (one run per side), the same switch costs 11.1 points (p =.27) and cuts output by more than half. The authors read thinking-dependence as a property of the model–harness pair and say the grid is too small to identify the mechanism.

Leakage in the discarded online runs#

Before the offline condition was enforced, DeepSeek and Qwen arms ran with network access. A tool-call audit found 312 of 439 trials had fetched the upstream fix (GitHub search, commit patch, PR diff or post-fix raw file), 152 of them citing the task's own issue or PR number, and 310 then edited a file. Those trials passed at 96% against 71% for trials that never touched the network. The Claude models did this in 4 of 445 trials. All web-enabled results were discarded, and the restriction moved from the prompt to the network layer. This is Evaluation-Time Answer Leakage's upstream channel on SWE-bench Verified, at the high end of that page's prevalence range.

Where it sits#

• Against Harness Tax: Coding-Agent Cost Multiplies Across Harnesses While Success Barely Moves (Arena.ai). Both find that harness choice barely moves success and moves cost by multiples. This paper adds the rerun yardstick that turns "±2–5%" into "indistinguishable from a rerun", plus equivalence bounds. It disagrees on two cost details. Arena's Claude Code is the most expensive harness in every shared-model comparison, while here, on DeepSeek-V4-Flash, Claude Code costs 0.95× mini-SWE-agent at DeepSeek's prices. Arena's cleanest pair has matched turns, while here the harness also moves the step count (OpenCode's 28 vs mini's 71 calls). The two are compatible: Arena's per-turn prefix gap is this paper's preamble term, and the step-count term only shows when a harness changes how long the model runs. Arena's Claude Code first call (27.0k tokens, 23 tools, Anthropic models) is larger than this paper's 17.1k. Here Claude Code runs against non-Anthropic endpoints, where its Grep tool is unavailable.
• Against Cache-Aware Agent-Loop Cost (TurnCost) (TurnCost). TurnCost simulates an ideal cache and gets 98.1–99.0% hits. Here provider-reported token splits on two production APIs give 96–99%. TurnCost finds turn count carrying 84% of log-cost variance within a harness. Here, across harnesses, the fixed preamble is the second multiplier. Prompt-Cache Economics is the one place in the corpus where cache price and prompt shape are optimized jointly. This paper shows a 9× difference in hit price (0.287× vs 0.033×) leaving the harness ranking unchanged.
• Against Orchestration Sets Token Economics (Writer). Writer's −41% cost came from one vendor's harness against its own superseded loop. This is the competent-third-party version, with three shipped harnesses on the same model and tokens at one price list: up to 3× in cost, at pass rates a rerun cannot separate.
• Against Tool-Catalog Context Cost. The 16,581-token Claude Code preamble against mini's 829 is another measured size of the resident per-call tax, and here it is the largest single driver of the cross-harness bill.
• For Compute-Controlled Benchmarking. Harness choice is an undisclosed budget variable that moves cost 3× while leaving score inside the rerun band, so a leaderboard mixing harnesses ranks bills as much as models.
• For The Harness Adaptation Gap. That study calibrates its own execution noise (ten runs of one snapshot, suite SD 3.99) before reading harness revisions. This paper is the same discipline applied to whole production harnesses.

Evidence weighting#

empirical, confirmed on a full read. It is the first harness comparison in the corpus with rerun pairs of identical cells, paired exact tests, an equivalence margin, power computed from the observed discordance, and every arm priced from its own token counts under one list. The analysis scripts, trial table and Harbor job files are public (github.com/YangzeLiu/what-does-a-harness-buy). No COI: the authors are academics, and none of the three harnesses or the HY4/GLM/DeepSeek/Qwen models is theirs. The paper carries an AI-use statement (generative AI used for implementation, analysis and drafting, reviewed by the authors).

Limits, most of them the authors' own:

• One benchmark. A one-model Terminal-Bench 2.1 transfer check (50 tasks: 19 / 20 / 20, all p = 1.0, flip rates 10–26%) is consistent but small. SWE-bench Verified is public and long-published, so contamination is possible and untested.
• The pool runs only two locally served Qwen models, so the ±5 equivalence holds for those two. The other three models have hard45 only, where nothing can be resolved.
• The closed frontier model runs in Claude Code only. Whether Opus-class models show the same gaps across harnesses is untested.
• Claude Code against non-Anthropic endpoints is not Claude Code as most users run it. Grep is unavailable (search goes through Bash), the HY4 arms use a proxy that folds mid-conversation system messages into the preceding user turn, and Claude Code sends an xhigh effort setting that vLLM ignores. The paper argues all three harnesses see the same thinking budget on the Qwen arms.
• Wall-clock limits were asymmetric on the Qwen arms (Claude Code 12× the task timeout, others 6×, OpenCode's own 5,760 s soft timeout binding). This is reported as uncontrolled.
• The reasoning re-send term is inferred from the token reconstruction, not observed.
• Unadjusted tests. McNemar p-values are not corrected for the number of pairs, and Holm's correction is applied only to the pool headline.

Connections#

• Harness Tax: Coding-Agent Cost Multiplies Across Harnesses While Success Barely Moves — the predecessor result (7 models × 3 harnesses, one run per cell), now with a rerun yardstick. The success finding survives it. The cost ranking does not generalize to every model, since Claude Code ties or undercuts mini-SWE-agent on DeepSeek-V4-Flash
• SWE-Bench Pro (Lineage and the V2 Reissue) — the other three-scaffold study on a SWE-bench-family benchmark, with mini-swe-agent and OpenCode in common. Its 0.29% scaffold share of outcome variance and its one loss (Inkling under Pi) match this page's "a swap is a rerun; effects are losses"
• Cache-Aware Agent-Loop Cost (TurnCost) — provider-reported 96–99% cache hits on two production APIs against TurnCost's simulated 98–99%, and the across-harness multiplier (fixed preamble) beside TurnCost's within-harness one (turn count)
• Prompt-Cache Economics — a 9× difference in cache-hit price that rescales the bill and leaves the harness order intact
• Orchestration Sets Token Economics — the configuration-level, third-party, same-model cross-harness token measurement that page's open question asked for, on coding
• Tool-Catalog Context Cost — the per-call preamble measured as a residual for three shipped harnesses: 16,581 / 7,025 / 829 tokens
• Every-of-k Reliability — run-to-run churn as a number: a rerun of an identical cell flips 13% of hard tasks, the same as a harness swap
• Agent-Authored Harness Optimization — the resolution arithmetic for reading harness gains: no gap reaches 80% power on 45 paired tasks, and 447 resolve 5.2 points
• The Harness Adaptation Gap — the same practice of measuring execution noise before reading a harness change, applied to self-built ARC-AGI-3 harnesses
• Compute-Controlled Benchmarking — the harness as an undisclosed cost axis: a 3× bill at a score a rerun cannot separate
• Evaluation-Time Answer Leakage — 312 of 439 networked DeepSeek/Qwen trials fetched the upstream fix (96% vs 71% pass), against 4 of 445 for Claude models
• Harness Shrinkage as Models Improve (hub) — a harness that strips most of the scaffolding (mini-SWE-agent, one bash tool, 829-token preamble) is statistically equivalent to Claude Code on pass rate, with the cost then decided by how many steps the model takes
• Claude Code — the heaviest preamble measured here (16,581 tokens), the only harness of the three that re-prompts after an output-token cut, and an Opus 5 grid where tool ablations and a year-old release move cost but not score
• Trace-Detector Measurement Validity (Silent Tool-Output Override) — a budget-only ablation on another benchmark that clears its own rerun gap: on fault-injected GSM8K, raising the token cap from 1,024 to 4,096 moves nemotron-3-ultra from 21–24/30 to 25–27/30 at every step setting, while a repeated baseline cell differs by one (20 vs 21/30) and gpt-oss-120b stays at 27–29/30. Like OpenCode's output cap here, the effect is runs a limit censors and the benchmark scores as wrong
• Accessibility Profiles (Breadth vs. Reliability Across Rewordings) — the same rerun yardstick applied to wording: a meaning-preserving rewording flips more answers than a new sample in 11 of 12 settings, and a greedy rerun of one wording moves 5.5–11.6% of GSM8K inquiries, the floor against which its model-change counts are read

Open Questions#

• Does "a harness swap is a rerun" hold for a closed frontier model across harnesses? The three-harness ladder stops at HY4-Preview, and Opus 5 runs only in Claude Code. Settler: a frontier API model (Opus/GPT/Gemini class) run in at least two shipped harnesses with rerun pairs on a ≥447-task paired pool.
• Is thinking-dependence a property of the model–harness pair? Switching thinking off costs mini-SWE-agent 11.1 points on Qwen3.6 (p =.04) but not the other two harnesses, and costs Claude Code 11.1 points on HY4-Preview (one run per side). Settler: thinking on/off across all three harnesses on at least two models, with k ≥ 2 runs on the 447-task pool.

Sources#

• What Does a Harness Buy? Tokens, Mostly — Yangze Liu & Zhongyi Han (Shandong University; first author's listed email is illinois.edu), What Does a Harness Buy? Tokens, Mostly, arXiv 2610.04433, 2026-10-03, 22pp, empirical. Cited for the abstract, §1, §3 (setup), §4.1–4.5, §5, and Appendices A, B, D, F, G, H, I, J, K, L, with Figures 2 and 4 viewed. Table-parse warning: verify flagged table-collapse on Table 1 (paired comparison on pool447). The docling grid welds the three Qwen3.6-35B-A3B pairs into one row and shifts the Qwen3.8-27B rows (the CC − mini label lands under Qwen3.6, and the b/c and p cells are displaced). The table above was recovered from pdftotext -f 5 -l 5 -layout on the local PDF and reconciles with the §4.1 prose (−1.8, −1.4, +4.7, +7.9 to +9.2) and Table A7's baseline column. Table A3 also collapses two cell pairs (Qwen3.8 pool447 CC/mini, HY4 hard45 CC/mini), and their values match Table 1 and Table 2. Table 2's header is welded but its rows agree with Table A3. Table A7's superscript n values are merged into the cells (-9.3 43 is −9.3 at n = 43) and were read that way. Table A4 and Table A5 are clean against the prose