The Harness Adaptation Gap

Sources#

• Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3
• Harness Evolution Hits a Ceiling: When Weight Training Should Begin
• RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

Summary#

A model that can build a good harness is not thereby a model that can improve one. Jens Lehmann (TU Dresden; Amazon, work done outside Amazon) and Sahar Vahdati (Hannover / TIB) test the second capacity directly: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3 (arXiv 2610.30499, 2026-10-21, 25pp, empirical).

They name the quantity this wiki's Agent-Authored Harness Optimization page has been measuring one instance at a time. The harness adaptation gap for revision t is

G_t = [J_dev(H_t) − J_dev(H_0)] − [J_test(H_t) − J_test(H_0)]

This is the improvement on the tasks that produced the feedback minus the improvement on tasks that did not. A positive G means the loop looks better to itself than it is.

The headline numbers:

• Construction works. Every one of 27 planned builds qualified. Initial held-out RHAE ranged from 0 to 97.14 (mean 24.54).
• Revision mostly doesn't. Two revisions raised mean held-out RHAE by +2.80 (95% CI [−1.44, 7.35]), while development RHAE rose +8.42. The gap G₂ is 5.62 [1.78, 9.77], the one interval in the paper that clears zero. Sign counts are 15 gains and 12 losses (p =.70). At a five-point threshold that becomes 9 gains, 3 losses and 15 indeterminate.
• The models never abstain. Retaining the parent was allowed, yet every revision changed the implementation. A post hoc rule that keeps whichever snapshot scored best on the five development games gets +4.53 over H₀ and +1.72 [0.38, 3.67] over always deploying H₂.

The paper's own framing is about RSI risk analysis: a system's future performance depends partly on whether it can improve the software it acts through. The answer here, with weights and the coding bootstrap both fixed, is "it can build it, and it cannot yet reliably improve it."

The protocol: every descendant, no selection#

What separates this from the rest of the harness-evolution literature is that nothing is selected. Each lineage gets exactly two prescribed revisions, and every frozen snapshot is scored on all 25 public games. There is no archive, no best-of-N candidate search, and no validation-picked winner. This is the "every prescribed descendant" design that Appendix C contrasts with RRSI, Meta-Harness, AHE and HarnessOpt-Bench, all of which select among candidates.

• One model profile, two harnesses. Each condition pairs a model with an effort setting and uses it for both building and playing. The outer loop runs through OpenCode, a fixed general coding harness. The inner loop runs through the task harness the model wrote. The bootstrap never changes, and the task harness is the object of adaptation.
• Minimal seed. The repository holds the game interface docs, an LLM client, pricing docs and a run.sh contract. It has no solver architecture, no game rules and no worked example. The build prompt (Appendix B, reproduced verbatim) asks for high RHAE "while using inference economically."
• Qualification, not scoring, gates a stage. A candidate must run and interact on two development games within a 50-action cap. It has up to ten repair cycles and faces no score threshold. Of 81 stages, 46 qualified raw and 35 after repair.
• Feedback split. There are five development games picked at random beforehand (tr87, vc33, r11l, bp35, ft09). The other twenty public games are strictly held out and influence nothing.
• Scale. Nine model–effort conditions across seven families, three independent lineages each: 27 lineages, 81 snapshots and 2,025 game evaluations. Every run was replayed against the official ARC service. The cost was at least $18,129 at frozen list prices, of which $16,130 was gameplay and $1,352 the coding loop.

The paper also splits the improvement capacity into four parts: (i) construct an executable harness, (ii) repair interface failures, (iii) map execution evidence to a concrete edit, and (iv) select edits that help on tasks the feedback did not cover. Its finding is that the first three are present in every episode and the fourth is not. All 54 revision episodes inspected item-specific evidence, targeted an evidence-backed issue and passed their own checks, yet held-out outcomes still went both ways. This is the same decomposition as Lin et al.'s "updating is not benefiting," reached on a different benchmark with the updater and the user being one model.

Results by condition#

Condition means, held-out RHAE, three lineages each. The four matched-effort rows come from the paper's Table 4. The rest are recomputed from Table 3 after recovering its collapsed cells (see Sources). The recomputation reproduces the pooled 24.54 → 27.34 and the 15-gain count exactly.

Four things in it matter.

Construction strength dominates everything revision does. The spread between conditions at H₀ (0.09 to 82.92) is roughly thirty times the mean revision effect. GPT-6 Astra's first build hit 97.14 on one replicate, and its weakest replicate is the paper's largest single gain (58.03 → 92.62). The paper's comparisons between conditions are descriptive only, three runs per cell.

Replicates disagree more than conditions do. All three Sol-high lineages took the identical strategy path (reorient, then extend). Their held-out outcomes ranged from +33.18 to −15.18. Sol-high replicate 1 went 20.38 → 29.24 → 53.56 at unchanged cost after raising an internal decision limit and adding animation-aware observations. Sol-xhigh replicate 1 went 43.95 → 21.99 while cost fell from $31.39 to $7.18 per game, after its first revision replaced response chaining with independent calls.

Weak builders get worse. Terra and Luna lose most of what little they had, which fits Agent-Authored Harness Optimization's headroom discriminator from the other end: a loop needs a harness that already works somewhat before revisions have anything to act on.

The gap is fragile to which games gave feedback. Dropping r11l from the development summary alone cuts G₂ from 5.62 to 3.02. Dropping any other development game leaves it between 4.68 and 7.84. A five-game feedback set is small enough that one game carries half the measured overfitting. At the game level, 13 held-out games improve and 7 decline on average, from +20.44 (lp85) to −15.03 (sb26). A small pooled gain is the sum of large effects pulling in opposite directions.

Noise is measured, not assumed. The authors re-ran one frozen snapshot, Sol-high replicate 1 at H₂, nine more times on the same twenty games. Suite RHAE had SD 3.99 (range 43.38 to 55.37), and the median within-game SD was 18.78. The mean revision effect (+2.80) is smaller than one run's noise. That is why the paper reports the ε = 5 threshold table and refuses to label individual lineages. The snapshot was picked post hoc and is one harness, so it is a scale for the noise, not a universal threshold.

Cost moves separately from score#

The harnesses got cheaper on average: mean logical cost per held-out game fell $10.30 → $7.57, but the median only $9.18 → $7.18. That average splits four ways at H₀ → H₂. Nine lineages were Pareto gains (better and cheaper), six gained score at higher cost, nine got cheaper by solving less, and three got worse on both. "The revision cut inference cost" is therefore not evidence that the revision improved efficiency. Half the cheaper harnesses had simply stopped trying as hard. Compute-Controlled Benchmarking argues this from the evaluator's side: score and spend must be reported jointly, and this paper does so per lineage, with a separate outer-loop cost.

The harnesses also grew in every lineage: median generated Python went from 2,161 to 2,604 to 2,912 lines, with no lineage replacing its harness family. Of the first revisions, 22 of 27 reorient the strategy (context continuity, replay, model-call cadence, budgeting), and the second revisions split 12/12/3 among reorient, extend and preserve. Edit size predicts nothing: large and small diffs fall on both sides of zero. This is the complexity-accumulation failure RRSI names, observed in a loop with no regularizer to stop it (see Agent-Authored Harness Optimization's ninth instance).

What the process coding can and cannot say#

All 54 episodes were packaged outcome-blind (model identity and downstream scores masked) and coded twice by GPT-5.6 Sol xhigh under a frozen codebook. Only three codes cleared the paper's κ ≥.67 bar: scope (.96), post-result adaptation (.84) and planning (.70). Planning was then dropped because half its packets were missing, all of them from the Opus and Sonnet conditions. Diagnosis (.35), incumbent protection (.47) and resource metareasoning (.29) were too unreliable to use. The two associations that survive both have intervals spanning zero:

• Broad rewrites vs contained edits: +3.75 vs −0.20 immediate held-out change.
• Reactive vs hand-off episodes: 18 episodes where a post-edit result triggered more edits, reverts or a scope change averaged −0.76. The 36 that linked the result to the change and handed off averaged +2.48. Reactive episodes also started from stronger parents (32.94 vs 21.33), so this is confounded.

The coders were never checked against human annotators (the paper's own Limitations). Treat this section as a well-built codebook with null results, not as a finding about which revision behaviors work.

Where it sits among the other harness-loop results#

Three controlled sources now speak to whether development gains survive on held-out tasks, and they reconcile on whether the loop has an acceptance gate and how large its feedback set is:

The paper's post hoc selection result is the bridge. Adding only a dev-score acceptance rule, with no new information, recovers +1.72 points over always deploying, and that interval excludes zero. The authors hold back from claiming it as a method because it was not specified in advance. The direction matches RRSI's ablation and Tian et al.'s gated loop: what transfers is decided more by the acceptance side than by the quality of the edits proposed.

On the RSI autonomy ladder this is L2 with the acceptance decision handed to the system. The ladder's L2 keeps the objective and the acceptance rule external. Here the objective is external, but the model alone decides whether to keep the parent, and it chose to change the harness 54 times out of 54. The updater itself (OpenCode plus the fixed model) is never revised, so the study says nothing about L5. The authors say so: claims about sustained acceleration "require longer runs, broader tasks, and updaters that can themselves change." For Recursive Self-Improvement the result is a clean null on the harness route under a two-step budget. It is not evidence that the route is closed, because the human-steered harnesses below show the headroom is real.

The headroom is real: human-steered systems#

At matched Opus 5 xhigh, Tycho reaches 100 RHAE on every public game, including the same twenty held-out games, against 39.95 for this study's revision-2 harnesses. VISTA and NVIDIA's AVO also report 100 with Opus 5, and Retrodict 99.86 with Sol at maximum effort. The development procedures differ: Tycho selected among four policies on all 25 public games using Opus 4.8. So this is attainable performance, not an isolated effect of human steering. Two within-study harness comparisons the paper cites make the same point more tightly. Schema reports 42.83 for a generic Claude Code baseline against 98.98 for its world-model harness (Opus 4.8 / Fable 5). Twin reports 61.1 for off-the-shelf Codex with Sol against 93.3 for its own harness.

Two calibration notes. OpenAI reports Sol at 38.3 on the public set in a simple benchmark harness with retained reasoning and compaction. That is above the 26.15–28.58 Sol-high's self-built harnesses reach on twenty of those games, although the game sets and effort differ. The Claude Opus 5 launch figure of 30.2 (high) is a different evaluation set and harness from this study's 11.71–15.43 for Opus 5 high, and the two should not be compared directly.

Evidence weighting#

empirical, confirmed on a full read, and stronger than most of the harness-evolution literature on three counts. It has a strict held-out split with no selection, so no winner's-curse inflation. It uses whole-lineage bootstrap CIs on the headline numbers (100,000 resamples), where RRSI and DarwinX report none. It measures execution noise and publishes the threshold sensitivity table. Every run is also replayed against the hosted ARC scorer, and the supplementary artifact includes all 81 snapshots and deterministic reproduction scripts.

The limits are the authors' own: one benchmark, five feedback games, two revisions, three runs per condition, and each snapshot evaluated once apart from the single noise re-run. The games are public, so contamination is possible, and the paper does not test for it; Astra's near-ceiling first build is where that would matter most. The process coding is LLM-only.

COI: Lehmann and Vahdati are authors of Tycho, the human-steered system they cite as the 100-RHAE ceiling. That bears on the headroom framing, not on the study's own measurements. No vendor affiliation applies to the models compared.

Connections#

• Harness Swap vs. Rerun Noise (What a Harness Buys) — the same measure-the-noise-first discipline on whole production coding harnesses: a harness swap flips as many SWE-bench Verified hard tasks as a rerun, and no paired gap on 45 tasks reaches 80% power
• Agent-Authored Harness Optimization — the home page for harness self-evolution, where this is the tenth instance. It is the first one that builds the harness from scratch, scores every prescribed descendant without selection, and states the dev-minus-held-out gap as a named, interval-bounded quantity
• Harness or Weights: Choosing the Lever by Failure Composition — the controlled counterexample on direction: with a 48-task pre-registered development split and a noise-gated accept rule, held-out gains match the dev gains. The table above reconciles the two by feedback-set size and acceptance gate
• RSI Autonomy Levels (B0–L5) — the paper sits at L2 with the keep-or-change decision internalized, and the system never uses it. That is evidence about what L2's external acceptance rule is doing, and it is silent on L5 because the updater never changes
• Recursive Self-Improvement — the harness route to self-improvement under a two-step budget: building succeeds, improving is not reliable. The authors' risk framing (a system that can improve its own software) is the same one that page's L2 discussion tests
• Intelligence Explosion Dynamics — a two-point capability-per-improvement series beside dollar spend, which is the shape of the data that page's returns question needs, but each step's gain sits below a single run's noise
• Compute-Controlled Benchmarking — score and cost reported jointly per lineage, with outer-loop spend separate, and the reason it matters: nine of the cheaper harnesses got cheaper by solving less
• Harness Activation and Adherence — Lin et al.'s update-versus-benefit split, which this paper's four-capability decomposition reaches independently
• Harness Shrinkage as Models Improve — the opposite direction: every lineage here grows its harness (median 2,161 → 2,912 lines) whether or not the growth helps
• Global Proof Coordination — the same human-designed-harness headroom in a kernel-verified domain: Maestro over plain Claude Code / Codex on the same backbones, largest on research-level Lean targets
• GPT-6 Astra, Claude Opus 5, Claude Fable 5, Claude Sonnet 5 — the models whose self-built harnesses are compared

Open Questions#

• Does a prospectively specified acceptance rule (keep the parent unless development RHAE improves beyond a calibrated noise band) reproduce the post hoc +1.72 over always-deploy, and does a larger development split shrink G₂ below its 5.62? The paper finds r11l alone carries half the gap and flags its selection result as post hoc. Settler: a rerun of this protocol with the rule pre-registered, or with ten or more feedback games.
• How much of the construction spread, especially GPT-6 Astra's 97.14 first build, depends on the games being public? The paper names possible training exposure as a limitation and runs no check. Settler: the same lineages scored on ARC-AGI-3's semi-private or private games.

Sources#

• Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3 — Jens Lehmann (TU Dresden; Amazon, work outside Amazon) & Sahar Vahdati (Universität Hannover / TIB), Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3, arXiv 2610.30499, 2026-10-21, 25pp, empirical. Cited for the abstract, §2 (G_t and the four capabilities), §4 protocol, §5 results, §6 strategy audit and process coding, §7 human-steered comparison, §8 limitations, and Appendices A–G (Tables 1, 3–14, 16–17; Figure 3 viewed). COI: authors of Tycho, the human-steered comparison system. Parse warning: PDF-derived (docling 2.126.0 / docling-mlx 0.1.1). Table 3 is collapsed in the raw parse: GPT-6 Astra replicate 2's revision-2 cell (92.62/11.96) is welded into replicate 1's row, and the Claude Opus 5 xhigh replicate 1 and 2 cells are welded into one row. Recovered from pdftotext -f 16 -layout on the local PDF, and the recovered table reproduces the paper's pooled means (24.54 / 27.34), its 15-gain count and all four Table 4 condition means. Every other table checked against the prose and consistent.
• Harness Evolution Hits a Ceiling: When Weight Training Should Begin — Tian, Hu et al., arXiv 2610.11655, empirical. Cited only for the reconciliation row (48/72 pre-registered split, held-out gains). Full treatment on Harness or Weights: Choosing the Lever by Failure Composition
• RRSI: Regularized Recursive Self-Improvement of Agent Harnesses — Xia, Han et al., arXiv 2609.24972, empirical. Cited only for the reconciliation row (unregularized vs regularized ablation). Full treatment on Agent-Authored Harness Optimization