Agent-Authored Harness Optimization

Sources#

• DarwinX: Evolving Agent Harnesses Through Natural Selection
• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
• How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes
• Knowledge-Centric Self-Improvement
• Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
• Recursive Self Improvement for Coding Agents
• Rethinking the Evaluation of Harness Evolution for Agents
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
• Sidekick's continual learning loop

Summary#

The loop in which an agent optimizes the scaffold it runs inside. Give a coding agent (a) a benchmark it can run, (b) write access to the harness source, and (c) a goal expressed as a score, and it executes the whole eval-fix cycle unattended: run the baseline, read failure traces, form hypotheses, patch the harness, re-run, keep or discard, log what it tried. This is the autonomous form of the loop Google's flywheel deliberately keeps human-gated, and it sits one level above Agent Harness Engineering — the harness patterns are the object being edited rather than the thing a human authors.

The corpus holds three instances, and they do not agree:

• Cline's July 2026 campaign (Cline, case-study) — the loop running in the wild. One prompt to GPT-5.6-Sol as leader model, 17 hours of continuous autonomous work, ~1B tokens (400M agent + 600M eval reruns) and ~$680 total, producing a merged PR against Cline's own production repo. Weight it as a vendor build log, not a measurement — see Evidence weighting below.
• HarnessBank (Luo et al., arXiv 2607.13683, empirical) — the loop under laboratory control. Sealed per-domain test splits, two competing self-evolution methods on a matched rollout budget, a four-check statistical screen on every candidate, and an ablation of its own selection rule. Six of seven benchmarks post credited held-out gains of +9.2 to +15.4pp.
• Wang, Zhu, Hu et al. (Ai2 / UW, arXiv 2607.12227, empirical) — the loop against budget-matched baselines, which neither of the other two ran. Give parallel sampling, sequential refinement and harness evolution the same feedback and the same K = 5 inference budget on Terminal-Bench 2.1, and harness evolution loses to the simplest baseline and does not transfer to held-out tasks.

The third is the one that changes the reading of the other two, so it comes first below. Cline remains the only instance anyone has run against a real production harness.

Two further instances arrived in August: Ouroboros (case-study, unbriefed and unmeasured) and DarwinX (empirical), which reports the opposite of Wang et al. on Wang et al.'s own benchmark. The discriminator that fits all five is how broken the starting harness was — developed in The fifth instance below.

The control nobody ran: budget-matched baselines#

The methodological charge is one sentence, and it applies to every prior result on this page: harness evolution is itself a search procedure that repeatedly evaluates and revises candidates using task feedback, so it has to be compared against methods that spend the same budget searching over trajectories instead. Otherwise a reported gain cannot be attributed to better harness design rather than to more test-time compute — this is Compute-Controlled Benchmarking's benchmark-maxxing critique aimed at a named method class for the first time, with the experiment attached.

The second charge is narrower and just as damaging: prior protocols search and report on the same public benchmark, so gains "may reflect adaptation to task-specific patterns rather than improvements in harness design."

The setup#

Terminal-Bench 2.1 (89 tasks), Claude Opus 4.6 / GPT-5.4 / GPT-5.4 mini, high reasoning effort, 128k max generation, every result averaged over two independent runs. All four arms start from an identical minimal harness: one bash tool, no skills, no middleware, no persistent memory. Budget fixed at K = 5 for every method, with AHE (Lin et al.) sampling m = 1 rollout per task so the rollout counts match. The four arms differ only in what the budget buys:

Arm	Budget spent on	Object updated
Parallel sampling	K independent trajectories	nothing (fixed harness)
Sequential refinement	K conditioned revisions of one trajectory	nothing (fixed harness)
Harness evolution	K rounds of meta-agent harness edits across a task batch	a shared harness
Harness scaling	K rounds of harness edits for one evaluation instance	a per-task harness

Harness scaling is the paper's own construction and the sharpest instrument on the page: it is harness evolution stripped of its reusability claim — instance-guided rather than dataset-guided — and it exists precisely to ask whether the gains come from reusable design or from spending more compute at test time.

One honest scoping note: harness evolution is instantiated as AHE with its explore agent disabled, so that gains cannot come from retrieving benchmark-fitted harnesses off the shelf. That makes the comparison fair and also means this is a negative result on one method, not a replication failure of HarnessBank, which was never run.

Without unit tests, the simplest baseline wins#

pass@1, Terminal-Bench 2.1, self-judge selection for the parallel arm (Table 1; independently confirmed by Figure 1's averages):

Method	Claude Opus 4.6	GPT-5.4	GPT-5.4 mini	Average
direct sampling, initial harness	69.9	75.3	59.4	68.2
Parallel sampling	74.7	79.2	62.9	72.3
Sequential refinement	73.0	73.0	61.8	69.3
Harness evolution	71.4	69.7	61.3	67.4
Harness scaling	76.0	78.1	61.2	71.8

Harness evolution is the only arm that finishes below the do-nothing baseline (67.4 vs 68.2), and on GPT-5.4 it costs 5.6 points outright (75.3 → 69.7). Parallel sampling — draw five, let the model pick — gains on all three models. The paper's reading is that self-generated feedback is noisy and sequential revision compounds early mistakes, so harness revision needs an external correctness signal to be grounded at all.

With unit tests, harness evolution's gain is multi-attempt selection#

Give every arm oracle unit-test feedback and oracle selection, so pass@5 becomes meaningful (Table 2):

Method	Opus 4.6 pass@1	pass@5	GPT-5.4 pass@1	pass@5	Avg pass@1	Avg pass@5
direct sampling, initial harness	69.9	—	75.9	—	72.9	—
Parallel sampling	84.8	84.8	87.1	87.1	86.0	86.0
Sequential refinement	83.1	90.4	85.4	93.3	84.3	91.8
Harness evolution	73.0	83.2	78.6	89.3	75.8	86.2
Harness scaling	83.1	89.9	82.0	88.8	82.6	89.3

Everything improves over direct sampling — unit-test feedback helps whatever consumes it — but harness evolution is last on pass@1 by more than 10 points and last on pass@5 too. The inference the authors draw is the one worth keeping:

If harness revision genuinely produced better harnesses, we would expect the improvement to be reflected in pass@1. Instead, the benefit only materializes when we can select among multiple trajectories.

That is benchmark-maxxing restated as a diagnostic: a method whose gain lives in pass@k and not pass@1 is buying attempts, not capability. Harness evolution's pass@1 barely clears direct sampling (75.8 vs 72.9) while its pass@5 (86.2) sits in the same band as parallel sampling's flat 86.0.

The transfer test: +0.6pp#

Split Terminal-Bench 2.1 into 45 train / 10 validation / 34 held-out test tasks, evolve on train with unit tests, select on validation, report pass@1 on test (Table 3):

Method	Claude Opus 4.6	GPT-5.4	Average
direct sampling, initial harness	63.3	72.1	67.7
Harness evolution	64.5 (+1.2)	72.1 (+0.0)	68.3 (+0.6)

+0.6 points on average, and exactly zero on GPT-5.4. The other three arms are not applicable here — they scale at the instance level and produce no reusable artifact, which is itself the point: only harness evolution claims to output something transferable, and this is what the claim is worth on this benchmark.

Read that +0.6 against the paper's own noise floor. The same direct-sampling configuration on GPT-5.4 scores 75.3 in Table 1 and 75.9 in Table 2 — a 0.6-point spread between two identical baseline conditions. No error bars, standard deviations or significance tests appear anywhere in the paper, so the headline generalization gain is the same size as the only visible estimate of its own run-to-run variance. The conclusion survives (the direction of the argument is that the gain is absent, and an absent gain does not need a significance test), but nobody should quote +1.2 on Opus as a measured improvement.

What the meta agent actually did, and why it didn't stick#

The failure is not incompetence at the edit layer. §5.1 reports the meta agent making rational, well-motivated edits across three layers, escalating in a recognizable order: prompt-layer behavioral rules first (produce deliverables early, copy fragile state before mutating it, recheck constraints before finishing); then, when advisory text plateaus, runtime enforcement through middleware — turn-budget trackers, truncation of oversized tool outputs, finalization gates that block completion when deliverables are missing; then tool-layer fixes correcting misleading guidance and injecting recovery hints. That is the same "gains live in runtime and control flow, not prompts" verdict this page records from HarnessBank and Deterministic Pre-Execution Gates — arrived at again, and still not enough.

The diagnosis is the most portable sentence in the paper:

Most edits memorize fixes rather than distilling strategies. Much of this information is precisely what a competent agent can rediscover through exploration within a single rollout, so persisting it in the harness saves time on tasks the agent could already solve but rarely converts failures into successes.

Three consequences follow, and all three are visible in the tables. A stable core of hard failures — deep domain reasoning, constraints outside harness control — is untouched by accumulated knowledge. The growing volume of persistent prompt text introduces context bloat that offsets the remaining gains (Context Lifecycle Management). And the arm that does best among the harness methods is harness scaling, which does not even pretend to build a reusable artifact: it encodes this task's known bugs, file paths and command sequences into this task's next attempt. Appendix B's case studies are explicit about it — count-dataset-tokens is fixed by embedding the dataset's config, domain and column facts directly in the prompt; db-wal-recovery by prescribing the exact backup-before-open sequence for that database. Those are per-instance notes, not harness design.

The escape hatch the authors leave themselves, and it is a real one#

§5.2 does not claim harness evolution is worthless; it claims Terminal-Bench cannot see the difference, for two stated reasons: agents already score highly there, so the residual failures may be model limits rather than harness deficits; and "a minimal setup consisting of a shell tool and a basic prompt already suffices for most solvable tasks," so performance is bottlenecked by reasoning rather than scaffolding. The recommendation is a benchmark specification: test harness evolution only where (1) tasks leave substantial headroom and (2) performance depends heavily on the harness — specialized tools, skills, workflows.

Take that seriously and it is also the cleanest reconciliation with HarnessBank, below.

Three sources, one benchmark family, opposite verdicts#

The conflict is real and worth stating flatly before reconciling it. Same benchmark family (Terminal-Bench), and three different answers:

	Cline (case-study)	HarnessBank (empirical)	Wang et al. (empirical)
Benchmark	Terminal-Bench 2.1	Terminal-Bench 2 (among seven)	Terminal-Bench 2.1
Model	Kimi K3	Qwen3.6-27B	Opus 4.6 / GPT-5.4 / GPT-5.4 mini
Vanilla baseline	77.5%	36.1	68.2–72.9
Reported gain	+11.3pp	+9.3pp (held-out, z ≥ 1.96)	+0.6pp (held-out)
Held-out split	none	per-domain, sealed	45/10/34 within the suite
Budget-matched test-time-scaling arm	none	one domain only (16× tokens on LiveCode)	four arms, whole suite
Starting harness	mature production harness	each benchmark's vanilla harness	deliberately minimal (one bash tool)

(Now four answers. DarwinX — Salesforce, empirical, 2026-07-31 — reports base Monet 75.5% → 83.2% on the same Terminal-Bench 2.1 with a matched frozen GPT-5.5, a genuinely disjoint held-out arm, and no budget-matched baseline. It is the largest single change to this section since Wang et al. and gets its own treatment below; the tier reasoning and reconciliation in this section are updated for it in place.)

By evidence tier the ordering is not close. Two empirical sources with sealed splits outrank one vendor-claim-adjacent case-study (superseded 2026-08-13: three empirical sources now) in which the vendor ran the benchmark on its own harness and published its own scores. Between the two empirical sources, Wang et al. holds the control HarnessBank's own open question named as missing (a full-suite test-time-scaling arm) while HarnessBank holds the controls Wang et al. does not need (a significance gate, an archive ablation, seven domains rather than one). DarwinX, the third, holds the widest evaluation ladder (four regimes, an anti-cheating audit, 1,260 held-out real tasks) and not the one control that would settle its disagreement with Wang et al. — a budget-normalized arm. Vendor status does not separate them either: DarwinX is Salesforce evaluating Salesforce's own proprietary agent, which is Cline's structural position with an empirical protocol on top.

The reconciliation that fits all three is headroom and harness sensitivity — the authors' own §5.2, applied to the other two papers. (Sharpened 2026-08-13 by DarwinX: Evolving Agent Harnesses Through Natural Selection: score headroom is the wrong variable — Wang et al.'s baseline is 68.2, below base Monet's 75.5, and gets the smaller gain. What survives is the second half, harness sensitivity, restated as how broken the starting harness was — see the fifth-instance section below. The first two bullets that follow are unchanged and remain correct about each source; only the name of the discriminator moves.)

• HarnessBank's Terminal-Bench arm evolves a 36.1-point baseline on a 27B open-weight backbone. Wang et al. evolve frontier models already at 63–75. §5.2's first condition — substantial headroom — is satisfied in the first case and not the second. HarnessBank's own strongest gains sit on its weakest baselines (BrowseComp+ 16.9, AppWorld 41.3), which is the same pattern from inside its table.
• Cline's harness starts mature, and the five fixes its agent found were defects — a collapsed reasoning-effort setting, unretried 429s, a loop detector that couldn't read output, an async-worker crash, pkill self-termination. Repairing a broken production harness and improving an already-adequate minimal one are different tasks with different ceilings, and only the first was measured by Cline. Nothing in Wang et al. contradicts "Cline's harness had five bugs"; what it contradicts is the inference from that to "harness evolution is how you should spend your compute."
• Which leaves the sharpest unresolved piece: §5.2's second condition is untested by anyone. No source in the corpus has run harness evolution on a benchmark chosen for harness sensitivity (superseded 2026-08-13 by DarwinX: Evolving Agent Harnesses Through Natural Selection: WebArena-Infinity is that test, arrived at accidentally. Monet's browser harness was a coding agent pointed at Chrome — 43.5% audit-clean, 20.0% on one application, 23.5% of its successes invalid — so performance there depends on the harness about as heavily as it can. The evolved harness reaches 93.0% on 1,260 unseen real tasks. §5.2's second condition, satisfied, gives the largest gain in this page's corpus, which is the strongest support the condition has.), and Terminal-Bench may be the wrong instrument for the entire literature that reports on it.

What this does not license. It is not a refutation of HarnessBank — different method, different backbone, different benchmark version, and Wang et al. never ran HarnessBank's loop. It is a refutation of the protocol that every prior result was produced under, and its force is proportional: hardest on Cline (no held-out split, no baseline, self-scored), real but partial on HarnessBank (sealed splits satisfy the second charge; a single-domain 16×-token arm does not satisfy the first).

This is not "the harness doesn't matter"#

Worth stating explicitly, because the corpus's three other controlled harness studies all point the opposite way and none of them is threatened by this result. Orchestration Sets Token Economics holds the model fixed, swaps a hand-built orchestration layer, and moves cost −41% and tokens −38% uniformly across six models. Measuring Beyond Accuracy Saturation swaps the scaffold and moves accuracy ~44pp. Harness-Induced Belief Divergence swaps the evidence-mediation layer and moves the agent's beliefs. The harness is a large lever in all three.

The distinction is hand-built versus machine-evolved. Those three vary a harness a team designed, across dimensions a team chose. This varies a harness a meta agent derived from benchmark feedback, one round at a time, and finds the derivation does not pay for its own compute. Both can be true at once, and §5.1 says why: the meta agent's edits memorize this suite's fixes rather than distilling design principles, so what it produces is not the kind of object those three papers measure. A big lever pulled by a search process that cannot see where to pull it is still a bad use of budget.

There is a second-order note in the same direction. The middleware Wang et al.'s meta agent generates — output truncation, finalization gates, turn-budget interrupts — is drawn from exactly the classes Harness-Induced Belief Divergence measures as belief-shifting interfaces. An evolution loop therefore manufactures evidence-mediation changes as a side effect of chasing a score, and nobody has measured what that does to the agent's belief state.

The fourth instance: unbriefed, deployed, and unmeasured#

All three sources above share a protocol — a human writes a brief, a benchmark supplies the objective, the campaign terminates. Ouroboros/Hope (Razzhigaev et al., arXiv 2608.08311, case-study) breaks it: 161 continuous days of a deployed agent editing its own core, 1,085 self-modification commits at 94.2% agent-authored, where improvement is either scheduled as its own recurring task or triggered by ordinary work and user complaints, and every change passes a blocking multi-model diff-review gate (63.5% recent block rate). No brief, no score, no end.

Two things it settles for this page, and one it conspicuously doesn't.

The loop does run unattended for months without a human re-issuing a brief. That is the property Recursive Self-Improvement has been using as a separator and that HarnessBank's 10-round termination floor showed the gated benchmark loop lacks. So "it converges rather than compounding" is a property of the benchmark-objective protocol, not of self-modification as such — remove the objective and the loop keeps going.

And what it produces is still maintenance. The two evolution traces the paper follows end to end are a duplicate-send guard in the public output pipeline (surfaced by users noticing repeated messages) and a review-pack context-overflow fix replaced by an import-graph-centrality-ranked context atlas. Both are ordinary harness defects, and they are the same character as Cline's five fixes and the same character as Wang et al.'s §5.1 diagnosis: the loop repairs what is broken and does not distil design principles. Third independent arrival at the same verdict, from a protocol none of the others used.

What it does not supply is any measurement. The benchmark scores in the paper were produced on frozen seeds with self-evolution disabled (Appendix C says evolution off on four of five rows), there is no pre-evolution baseline harness anywhere, and the deployment's only time series plots spend, tokens, published LOC and memory artifacts. So the loop that runs forever is the one nobody scored, and the scores that exist came from a loop that was switched off. Read against the budget-matching charge above, that is not a rebuttal — it is the same critique with the measurement removed entirely.

One structural note worth carrying: the direction of the edit matches Cline's, not the shrinkage thesis. 175,755 published LOC and 1,085 commits, with the only net deletion in the final month. Agent-authored optimization grows the harness in every instance the corpus holds — and DarwinX below makes that structural rather than incidental: its edits are additive by construction ("because edits are additive, a branch accumulates capabilities rather than trading one for another"), so its selection rule cannot return a smaller harness even in principle.

The fifth instance: the same benchmark, the opposite verdict (DarwinX, July 2026)#

DarwinX (Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang et al., Salesforce AI Research / Agentforce, arXiv 2608.07545, 2026-07-31, empirical, 33pp) is the head-on contradiction: it reports large harness-evolution gains on Terminal-Bench 2.1, seventeen days after Wang et al. reported none there, and it supplies the held-out arm Wang et al. named as missing. Both are empirical; neither is averaged away below.

The method is the archive argument taken further than HarnessBank takes it. The base model is frozen; the harness spans a skill layer (prompts, memory, distilled knowledge) and a code layer (tools, control flow, agent loop). A child is admitted under a preserve-and-extend contract — net gain g(c) = Σ Δₜ > 0 and bounded regression R(c) = Σ(−Δₜ)⁺ ≤ δ, adjudicated by a verifier agent, then re-tested at higher avg@k with a preservation probe before it may steer search. Losing variants are retained as recombination stock, and complementary specialists are merged, the merged child kept only if it covers the union of its parents' solves. The design separates exploration (permissive admission on a noisy signal) from confirmation (strict avg@k before a variant is trusted) — which is HarnessBank's paired-2σ gate restated as a two-speed rule, plus the cross-lineage merge operator DGM and HarnessBank both lack.

Four regimes, ordered by how far the evolution signal sits from the test:

Regime	Base	Evolved	Gain	What controls it, and what doesn't
TB2.1 in-domain (89 tasks, avg@5, GPT-5.5)	75.5 ± 3.5	83.2 ± 1.2	+7.7	matched model; search set = report set; base row at default effort against the evolved row at high
TB2.1 leaderboard (GPT-5.6 Sol)	81.8 (OpenAI's own single-agent reference)	84.7 ± 1.2	+2.9	the paper's only tier-matched pair (same model, both medium) — and its smallest headline
TerminalWorld held-out (41 disjoint tasks, pass@1, Opus 4.8)	61.0	68.3	+7.3 (3 tasks)	genuinely held out, and McNemar p = 0.45 against its own base, p = 1.0 against Claude Code
WebArena-Infinity (1,260 unseen real tasks, GPT-5.5)	43.5	93.0	+49.5	task distribution and reward source both change; n = 1,260; deterministic verifiers
SWE-bench Verified (transfer, 500 issues, Opus 4.8)	—	84.2	+3.4 vs an LSP fix-skill reference	no matched-model base arm exists; every compared harness lands in 80.8–84.2

(Table 2's row set is the paper's, verified cell-for-cell against the PDF at ingest; the effort column is the paper's own and is what the next subsection is about.)

The budget control is a vendor label, not a matched budget#

DarwinX was the corpus's most promising candidate for the control budget-matching charge demands, because §4.1 is titled "the gain is the harness, not compute" and offers what it calls "an effort-controlled comparison." Full-text verification says it is not one:

• "medium / high / xhigh" is never defined. Not in the body, not in Appendix B's per-benchmark protocol table — which has no compute column at all. No token count, no turn count, no wall-clock bound is attached to any tier anywhere in 33 pages, and no dollar cost appears in the paper.
• So §4.1's control compares categorical tier names across different agents and vendors. Its argument — "a neutral harness at higher effort (Terminus-2, xhigh) reaches only 78.0%, below DarwinX's 83.2% at high effort, so raw effort in another harness does not reproduce the gain" — is a valid statement about tier escalation inside a different harness. It is not a normalized budget, because a tier name does not fix turns or tokens across harnesses.
• The headline matched-model pair is not itself effort-matched. Monet (base) is listed at GPT-5.5 / default (75.5%) against Monet (DarwinX) at GPT-5.5 / high (83.2%) — and that pair is the +7.7-point headline. The confound runs in DarwinX's favour on exactly the comparison the paper calls load-bearing, and against it on the Terminus-2 and Claude Code rows (both at xhigh against DarwinX's high/medium).
• §9 concedes it. "The archive, parent selector, recombination operator, and inference effort are not independently randomized."
• Its own compute figure shows the spend rising. Figure 5's medians on TB2.1: on the six newly-solved tasks the evolved harness runs 22 turns against 11 and 380K tokens against 89K; on the 69 both agents already solve, turns move 13 vs 12 — and tokens move 125K → 172K, a ~1.4× rise the prose does not mention while asserting that compute on solved tasks "barely moves" and that the harness "leaves solved tasks essentially untouched." (The 125K/172K pair is legible only in the figure; read from under the image two-pass rule.)

So DarwinX's answer to the budget charge is not "we spent the same" — it is "our extra spend is well-allocated." That is a different and weaker claim, and it is precisely what Wang et al.'s harness scaling arm was built to test: give one evaluation instance K rounds of harness edits and it is the best of the harness methods while producing nothing reusable. DarwinX does not defeat 2607.12227's critique; it instantiates it — it reports in-domain (charge 2: search and evaluation share one benchmark), with no budget-normalized arm (charge 1), on the benchmark whose §5.2 the earlier paper had already conceded may be the wrong instrument.

The held-out arm is weaker than the abstract#

TerminalWorld is the arm that should answer "does it generalize?", and by the paper's own reported statistics it does not answer it decisively:

• The gain over base is three tasks (25 → 28 of 41), McNemar p = 0.45. One solve is worth 2.4 points on a 41-task split, which the paper states.
• The margin over the strongest off-the-shelf agent is one task, McNemar p = 1.0, and the paper calls it "suggestive rather than statistically decisive."
• On the other base model the evolved harness loses to a neutral one. The same procedure on GPT-5.5 reaches 56.1% against Terminus-2's 61.0% — so Opus 4.8 is reported as the headline and the base where the harness underperforms a neutral baseline is reported as a "robustness caveat." That is HarnessBank's cross-model matching law showing through DarwinX's own numbers.
• Appendix C fixes the scope harder than the body does: "a separately skill-bundled pre-TW reference also reaches 28/41," i.e. a harness that never went through TerminalWorld search matches the headline exactly. The paper draws the right conclusion itself — the benchmark demonstrates that a diverse archive recovers a harness beating off-the-shelf agents, "not that TW-specific search lifts every possible starting harness."
• Figure 6 adds what the prose omits: of the four specialists the archive contributes, Specialist A (24/41 = 58.5%) is below the unevolved base (25/41) and Specialist B ties it. Half the population that gets merged into the winner does not beat doing nothing.

The one arm that genuinely contradicts Wang et al. is WebArena-Infinity, and it is a strong one: n = 1,260 real tasks never seen by the loop, evolved on 300 synthetic intents scored by an LLM judge and reported under deterministic verifiers, every one of ten applications improving. Raw-to-raw the move is 53.0 → 94.4 (+41.4); the headline +49.5 uses audit-clean scores, which penalise the base for 120 invalid successes it is not credited for — so ~8 of the 49.5 points are the base's compliance rather than the evolved harness's capability, which the paper states plainly ("capability and compliance improve together") and quantifies (confirmed-invalid 23.5% → 1.4%; invalid trajectories 293 → 17, the residual all raw-state mutations in one application).

Two structural notes on that arm cut against reading it as a vindication of the archive: every recombination was reverted on WAI (the gate keeps 26 iterations and reverts 36, and "every merge is reverted, so the gains accrue along a short accepted primary lineage"), and the in-loop proxy saturates while the held-out truth does not — TerminalWorld's training subset goes 0.505 → 1.000 against 68.3% held-out, a 31.7-point proxy/truth gap with "the variant that best fits the proxy… not the best generalizer." That is Optimizer–Evaluator Decoupling's problem stated as a measured quantity, and it is the sharpest thing in the paper.

What resolves the contradiction: how broken the starting harness was#

Reading DarwinX's four regimes against the other four instances, one discriminator predicts the sign and rough size of every result in this page — and it is not baseline headroom in the score, which fails on the pair that matters most (Wang et al.'s baseline is 68.2, below base Monet's 75.5, and gets +0.6 held-out where DarwinX gets +7.7 in-domain).

The predictor is whether the starting harness carried identifiable, harness-side defects.

Starting harness	Its defects	Reported gain
Monet's browser harness on WAI — a coding agent pointed at Chrome, 23.5% of its successes invalid, scoring 20.0% on one application and 95.8% on another	severe and specific	+49.5
HarnessBank's vanilla harnesses on a 27B backbone, one pathology dominating 49–88% of failures	severe and diagnosable	+9.2 to +15.4
Cline's production harness: collapsed reasoning effort, unretried 429s, a blind loop detector, an async-worker crash, pkill self-termination	five real bugs	+11.3
Ouroboros/Hope in deployment: a duplicate-send bug, a review-pack context overflow	ordinary defects	unmeasured
Monet's mature TB2.1 harness	none named	+7.7 (effort-confounded), +2.9 tier-matched
Wang et al.'s minimal seed: one bash tool, no skills, no middleware — spare but not wrong for a terminal benchmark	none	+0.6 held-out

WAI's own table is the within-source version of the same law: the evolved harness lands in a tight 84.0–98.3% band across all ten applications (mean 93.0) while the base ranges 20.0–95.8% (mean 43.5). The gain per application is close to ceiling-minus-base, and where the base was already strong (Elation clinical records, 95.8) the gain is +0.9. So the +49.5 headline measures the unevenness of the starting harness, not a uniform capability lift.

That is the fourth independent arrival at this page's standing verdict, from the source that looked most likely to overturn it: the loop repairs what is broken and does not distil design principles. Wang et al.'s §5.1 diagnosis ("edits memorize fixes rather than distilling strategies… saves time on tasks the agent could already solve but rarely converts failures into successes") and DarwinX's largest result are the same phenomenon with opposite framing — one measures a loop with nothing to repair, the other a loop handed a badly mis-fitted harness. Neither source needs to be discounted for both to be true.

What DarwinX does add that nothing else in the corpus had: the first cross-suite transfer measurement (a TB2.1-evolved harness run unchanged on SWE-bench Verified, and on a different base-model family), the first archive whose merge operator is specified as a union-coverage acceptance test, and the first harness-evolution paper to run an anti-cheating audit as a first-class result rather than a prompt clause — a two-stage static-plus-LLM detector at 99.0–99.4% coverage, with the evolved harness's evaluation-plane, privileged-host and exploit classes going to zero. On Reward Hacking that is the strongest available evidence that preservation-based selection does not reward verifier-gaming: 370 rewarded TB2.1 trajectories, two flagged, one false positive, and the one confirmed shortcut a per-trial policy lapse (reading an answer from a task's own published README) that the same harness solves legitimately in three of the other four samples.

The sixth instance, and the only one that reports what comes after the plateau (Shopify, August 2026)#

Shopify's Sidekick account (McNamara & Mazza-Anthony, Shopify Engineering, 2026-08-05, case-study — first-party, unreplicated, no controlled arm and no held-out split, so it sits at Cline's evidence tier and below the three empirical sources) adds an instance whose numbers are useless here and whose shape is the most informative thing in it.

The setup is this page's loop with the sharpest available statement of why prompt tuning is not enough: the target "is already a production application, with dynamically assembled prompts, custom control loops, and bespoke orchestration spread across a large codebase. No single prompt determines its behavior, so prompt tuning reaches only a small part of the system. The optimization target is the entire harness: its prompts, tool definitions, and orchestration code." The loop is explicitly a Karpathy-autoresearch instance — propose an edit, evaluate against the metric, "keeps the change if the score improves; and discards it otherwise" — configured in one program.md granting edit rights to prompts/, tools/ and harness/ with the note "everything here is editable, including the harness."

Three things it contributes:

It runs the decoupling properly, by architecture and by construction. The metric is a separately-calibrated LLM judge that the optimizing agent has no write access to — its directories are prompts/, tools/, harness/, and the judge is neither. That is a materially stronger arrangement than Cline's prompt-clause invariant and than DarwinX's in-domain search-equals-report set. The countervailing weakness is a different one: the judge is a learned instrument at 80% raw agreement against an 83% human ceiling, so the optimizer cannot edit its scorer but can still climb its noise, and nothing in the account re-validates it mid-loop.

It is the only source that reports the loop terminating, and what the team did next. Every instance above ends at a converged score, a significance floor, or a shipped PR. Here: "once harness improvements plateau, we begin optimizing in parameter space." No plateau value is published — no score, no round count, no spend — so this is an assertion, not a measurement. But it is a production team's stated ordering, and it points the same way as this page's standing verdict from the other side: if the loop repairs what is broken rather than distilling design principles, then a mature harness has little left to repair and the returns should run out. Shopify says they did, and then moved the object of improvement rather than the search budget — Knowledge-Centric Self-Improvement carries that taxonomy and Agent Quality Flywheel the loop it continues into.

It is a sixth data point for the how-broken-was-the-starting-harness law, with the sign right and the magnitude missing. Shopify's starting harness was a live production system already serving merchants at up to 2,000 requests per minute — the most mature seed in the corpus by some distance — and the reported harness-stage outcome is a plateau, i.e. the smallest possible gain, consistent with the table above. It cannot be added to that table: no held-out split, no baseline number, no gain figure of any kind for the autoresearch stage in isolation. Recorded as directional support, not as a row.

The seventh instance: a human PR gate, and a red test before the fix (Bridgewater, July 2026)#

Bridgewater Associates's PAT (How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes, case-study, first-party conference talk, no numbers of any kind) is the loop deployed inside a hedge fund, and it contributes two structural details that no other instance on this page has, plus the corpus's weakest evidence for any of them.

The fix is preceded by a failing benchmark the agent writes itself. On a Teach press — or autonomously, from background agents mining completed investor conversations — the loop first "create[s] a benchmark that we expect to fail. This shows that we can reproduce this poor behavior." Only then does it "iterate on our context repositories or the harness itself until that benchmark passes," and only then re-run the rest of the suite to check for regression. Every other instance on this page optimizes against a metric that already exists; this one manufactures the discriminating test before touching the target, which is the difference between a score that improved and a defect that was reproduced and closed. It is also the cheapest available guard against the failure mode this page keeps naming — an optimizer climbing its scorer's noise — because a red-then-green test on a specific reproduced behavior is a much narrower claim than a rising aggregate.

The human gate sits at PR review, not at benchmark authorship. The loop's output is a Slack message containing a pull request with the proposed changes to PAT's context and harness. Compare the placement elsewhere: HarnessBank gates at a sealed split, Cline gates at nothing published, Ouroboros gates at nothing at all. Gating at the diff is the strongest practical control in the set — a human sees exactly what changed before it ships to hundreds of users — and simultaneously the least evidential one, because approving a diff is not measuring an effect. Nothing in the talk reports the acceptance rate, the reject reasons, or a single before/after score.

As a seventh data point for the how-broken-was-the-starting-harness law, it is unusable. The seed harness is a mature production system with hundreds of daily users, which predicts small gains — but no gain is reported, in either direction, so the row cannot be filled. What it does corroborate is Shopify's structural point one level up: the optimization target is "our context repositories or the harness itself", i.e. the whole system rather than a prompt, in a codebase where no single prompt determines behavior.

Evidence weighting: this is the least-controlled instance on the page. A talk, by the builders, about their own product, with a corporate disclaimer attached stating no guarantee that discussed benefits will be achieved. It is admitted here for its mechanism — the red test and the PR gate are transferable design choices — and for nothing about its efficacy.

The Cline campaign, in numbers#

Terminal-Bench 2.1, Kimi K3 via OpenRouter, stock Cline harness as baseline:

Run	Score	Cost
Baseline (stock harness)	69/89 — 77.5%	$79
Combined candidate	77/89 — 86.5%	$65
Confirmation	79/89 — 88.8%	$49.8

Cost fell as score rose, which is the mechanistically interesting part: most of the gain came from not wasting tokens on doomed retries, self-kills, and false loop-detector aborts. Cline's comparison points — Claude Fable 5 at $552 and GPT-5.6 Terra at $400 on the same benchmark — are not harness-controlled (see Compute-Controlled Benchmarking), so the price ratio compares one tuned harness against whatever configuration those numbers came from.

The five experiments the agent ran, in order:

1. Reasoning-effort abstraction — the harness silently collapsed K3's max reasoning effort to high. Correctness fix, no score credit (OpenRouter was already mapping to the only supported effort), but it unblocked everything downstream.
2. Rate-limit retries — five baseline failures were single OpenRouter 429s killing healthy sessions. Raised retry count with exponential backoff; all five flipped.
3. Output-aware loop detection — the loop detector killed agents legitimately polling long-running background work (same command, changing output). Made the detector read output, not just the command; two tasks flipped.
4. Async-worker liveness — one task exited in 7.6s with zero tokens: any prompt containing @a-style tokens triggered a file-mention lookup on an unreferenced async worker, and the process exited before the model was called. One-line fix, deterministic flip.
5. PID tracking to stop self-termination — two tasks ran pkill -f with a pattern matching the harness's own command line and killed themselves mid-task. Tool guidance to track PIDs; both flipped. Cline reports this fix reduced but did not eliminate broad-match kills.

Every fix is a general harness bug, not a benchmark-shaped special case — which is Cline's argument that no Reward Hacking occurred, and it is the strongest thing in the post.

The controlled version: HarnessBank (Luo et al., 2026)#

Everything the Cline campaign lacks — a sealed test set, competing methods on a matched budget, an ablation of its own selection rule — is what HarnessBank supplies (HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution, EverMind AI / Shanda Group, empirical). It is the corpus's first external, controlled evaluation of this loop.

The architecture separates four things Cline had fused:

• a task agent — a frozen backbone (default Qwen3.6-27B) executing benchmark tasks under the current harness;
• an evolver agent — a different model from a different vendor (Claude Opus 4.8) that reads execution trajectories, names the recurring failure, and writes offspring harnesses;
• a deterministic evaluator owning sampling, scoring, activation logging and the statistical tests;
• a Harness Gene Bank, a quality-diversity archive derived from MAP-Elites.

The harness is partitioned into an immutable kernel (evaluation, bookkeeping, self-evolution and interface-critical code) and a mutable surface the evolver may edit: prompts, injected knowledge, runtime control logic, configuration. Each domain has disjoint train/test splits and the test set is touched exactly once, after evolution ends.

The bank is indexed semantically, not by task. Each cell is a pair — where the patch acts (prompt, knowledge, runtime, config) × why it exists (a pathology inferred from failure traces, e.g. thinking-runaway). Harnesses hypothesizing the same pathology compete inside one cell and only the best survives; harnesses addressing different pathologies stay alive to be recombined later. Parent selection is quality-biased (argmax utility over the vanilla harness plus the bank), so the strongest lineage is exploited while the archive keeps structurally different solutions from being discarded — the paper's answer to search collapse, the tendency of greedy evolve-and-keep loops to converge on a narrow class of safe prompt edits. The keying choice is argued explicitly: an archive keyed on tasks "would preserve harnesses indexed by the very tasks used to select them, which overfits by construction."

Gated Harness Screening is this page's missing control, built in. Rather than running every offspring on the full training set, candidates face four sequential checks on a sampled subset:

1. Validity — sandbox crashes and verifier timeouts trigger repair-and-retry instead of being scored as agent failures.
2. Activation — every patch declares an activation spec and emits a deterministic beacon; a harness that never fires is "inert" and rejected however good its score looks.
3. Paired significance — task-level paired differences against the parent on identical tasks, requiring z ≥ 1.96 (two-sided 5%).
4. Gain — the estimated improvement must be positive.

(The introduction calls these four gates; §3.3 implements three multiplied indicators, folding the gain check into the significance gate as its Δ̂ > 0 factor. Same test, different bookkeeping.)

Only survivors get full-training-set evaluation and compete for a bank cell. The design principle worth stealing: the LLM may hypothesize freely, but credit comes only from the gate. The why label is explicitly "an LLM-assigned hypothesis, not ground truth" — on AppWorld the loop misdiagnosed a capability limit as a knowledge gap, and the gate rejected the resulting patch (0/24 → 0/24 on its own target tasks, p = 1.0). Because a wrong label only steers which candidates get tried, it costs a rejected candidate, never a bad harness.

Results, and what they cost the competition#

Seven domains, Pass@1 averaged over K = 3 attempts, measured on tasks the evolving agent never saw:

Benchmark	Test Pass@1 vanilla → evolved	Gain	Train→test retention
AppWorld	41.3 → 56.7	+15.4	86%
BrowseComp+	16.9 → 30.8	+13.9	86%
LiveCode	58.1 → 71.8	+13.7	65%
Omni-MATH	54.3 → 66.0	+11.7	92%
TB2	36.1 → 45.4	+9.3	148%
GDPval	43.7 → 52.9	+9.2	110%
SWE-bench	47.4 → 52.6	+5.1 (uncredited: n=26, z=0.78)	37%

Six of seven clear z ≥ 1.96 on held-out tasks (p from <10⁻⁴ to 0.033; AppWorld strongest at z = 6.44, n = 168), each a single held-out comparison, so the many candidate comparisons during evolution cannot inflate it. Pass@3 rises on every credited domain too (+5.5 to +15.4), so the harness expands the set of solvable tasks rather than only the per-attempt hit rate. The authors decline to treat the retention ratio as a test — it divides two noisy estimates — and rest the claim on the credited held-out gain.

Two baselines ran under the same protocol: same frozen backbone, same splits, same paired-2σ ruler, budgets within 2.1× on every domain (780–2,310 rollouts, with GEPA outspending HarnessBank on two domains and DGM on one). Across five sealed tests, HarnessBank is credited on four, DGM on one, GEPA on none:

Test Pass@1	Vanilla	HarnessBank	GEPA (prompt-only)	DGM (ungated)
AppWorld	41.3	56.7	44.0	52.0
LiveCode	58.1	71.8	58.1	59.8
Omni-MATH	54.3	66.0	54.3	53.2
BrowseComp+	16.9	30.8	19.0	21.2
SWE-bench	47.4	52.6	47.4	56.4

Both baselines fail in ways this wiki has been circling:

• GEPA, prompt-only, ships vanilla in three of five cells. It finds no variant beating its seed on LiveCode in 47 iterations, because "thinking-runaway is not prompt-addressable"; on AppWorld, where the win surface is partly prompt-expressible, its training gain washes out on the sealed test (+2.8, z = 0.97). The gains in this problem class do not live in the prompt — a direct rebuke to the assumption that harness optimization is mostly prompt optimization.
• DGM, openly self-modifying but ungated, deploys a regression. It clears the bar once (AppWorld +10.7, z = 3.00). On LiveCode it selects its best of 15 generations from a 15-task K=1 spike (0.733, regressing to 0.533 on re-evaluation) and lands uncredited (z = 0.66); on Omni-MATH the harness it ships is worse than vanilla (-1.1%). This is what "keep what appears beneficial" buys without a significance gate.

The paper reports against itself where it should: on SWE-bench, DGM's raw 56.4 is the highest number in the table, and at n = 26 the gate credits neither method. Note also that HarnessBank's authors re-ran both baselines under their own protocol — standard practice, and the standard caveat.

The ablation: what the gate actually buys#

Ablating the paired-2σ gate on TB2 separates three things it might be doing (rows below the first are deltas against it):

TB2 configuration	Test Pass@1	False elites	Rounds to stop
HarnessBank (K=3, 2σ)	45.4	0	10.0
w/o 2σ	±0.0	+2	>20 (cap)
w/o confirm + 2σ	-1.6	+3	>20 (cap)
Vanilla	-9.3	—	—

Deployment is unchanged — train-argmax already picks the winning mechanism — so on this domain the gate is not buying the headline score. What it buys is the archive and the stopping rule. Without it two noise mechanisms enter as elites (one of them inert, its activation beacon never firing), and false elites then seed parents, spending future budget on noise. Termination is decisive: under single-run or K=3-mean crediting, phantom progress appears in 62–76% of post-convergence rounds, so the loop never meets its stop condition and runs to the cap, while paired-2σ stops at the 10-round floor. An unverified self-evolution loop does not merely risk a bad harness — it cannot tell that it has finished.

Two supporting notes: accepted edits span all four levers rather than prompts alone, and on most domains the credited harness stacks mechanisms drawn from more than one cell, with the per-cell elite being what keeps the second mechanism alive long enough to be recombined at all. And train-selection is a lower bound on what generalizes — on GDPval a variant ranked below the winner on train scored highest on test (+11.5% vs +9.2%), which is the argument for crediting on held-out data rather than trusting the training rank.

The evolved harness is a correction fitted to one model#

The most consequential result is a negative transfer one. Cold-started on other backbones the loop evolves different harnesses, each targeting that model's dominant pathology, and the patches follow a pathology → patch matching law (test ΔPass@1; bold = matched patch):

Domain	Model	Dominant failure	Patch A	Patch B
AppWorld	Qwen3.6-27B	empty "engagement" turns	verify-finalize +15.4	submit-verify +1.2
AppWorld	397B	careless errors	verify-finalize +0.2	submit-verify +13.6
AppWorld	Gemini 3 Flash	careless errors	verify-finalize +5.8	submit-verify +13.5 (z=4.15)
Omni-MATH	Qwen3.6-27B	thinks too much	27B stack +11.7	raise-reasoning +1.7
Omni-MATH	397B	runaway (shared)	27B stack +11.0	raise-reasoning +0.7
Omni-MATH	Gemini 3 Flash	thinks too little	27B stack -1.5	raise-reasoning +15.3

Three readings:

• Off the matching diagonal the patch is near-zero, and the wrong direction is harmful. Gemini shares the 397B's AppWorld pathology and reproduces the careless→checklist match across model families; on Omni-MATH the two Qwen generations share thinking-runaway and the 27B-evolved stack transfers nearly loss-free (+11.0 transferred vs +11.7 native). But Gemini reasons too little, so the transplanted recovery never fires (-1.5) while its matched patch turns the same lever the other way to +15.3. Turning it the wrong way costs -15.7 when stacked on the evolved 397B harness. Two model families need one lever turned in opposite directions.
• This is a real failure mode of the frozen model, not a scoring artifact. On five domains a single pathology dominates 49–88% of vanilla failures, and the matched mechanism is credited under both deterministic and judge-based verifiers.
• A test-time-scaling arm exists, and loses. On LiveCode neither a 16× token budget nor a blanket thinking-off toggle reproduces the evolved harness's gain — the credited mechanism disables thinking only after a runaway. On this domain, spending more compute is not a substitute for the specific control-flow fix (Large-Scale Test-Time Compute).

So the transferable asset is not the harness. It is the diagnose → search → verify process: "a credited harness is a correction fitted to the model, not a universally good setting."

The contrast that explains it: knowledge transfers where harnesses don't#

A second empirical source, published six days after HarnessBank's v1, reports the opposite transfer result for a different persistent artifact. Knowledge-Centric Self-Improvement (Wang et al., Caltech, arXiv 2607.19592) freezes a curated knowledge base rather than a harness, hands it to a fresh model family with no further curation, and gets a positive lift in all eight donor-recipient cells on held-out tasks, in both cross-family directions (Polyglot 8.3% → 20.0%, ARC-AGI-1 23.3% → 43.3% for the strongest pairing).

Both results are held-out, both are single-lab preprints, and neither is wrong. The reconciliation that fits both is what the artifact is fitted to:

• A harness patch is fitted to one model's failure distribution. That is why the same reasoning-budget lever has to be turned down for a runaway backbone and up for an under-thinking one, and why transplanting it costs -15.7. The matching law above is the direct evidence.
• A distilled insight is fitted to the task domain — an ARC grid invariant, Rust's loop-based propagation semantics, a test-runner flag. Those are true independently of which model reads them, which is why a Haiku-authored bundle helps GPT and vice versa.

Stated as a design rule: fit your persistent artifact to the problem, not to the solver, if you want it to move. Nobody has run both artifact classes under one protocol, so this is a hypothesis with two strong one-sided data points rather than a measured law — the open question lives on the sibling page.

The two papers also agree, independently and on disjoint benchmarks, about the weakest axis. HarnessBank credits GEPA on zero of five sealed tests; Wang et al. put GEPA last of three on both benchmarks they ran it on (44% and 36% against 86.7% and 68.0%). Two labs, two different objects of improvement, same verdict on prompt optimization.

The Caltech paper is also the sharper contrast on cost: its knowledge-curation runs beat DGM and HyperAgents on solve rate while spending less — SWE-bench Pro at $208 against DGM's $713 — where harness evolution buys its gains with a rollout budget in the hundreds to low thousands per domain.

Reconciling Cline and HarnessBank#

(The third source's disagreement with both is handled in Three sources, one benchmark family above; the two agreements below survive it, and the first is now qualified.)

• They agree on magnitude — against a baseline neither ran. Cline's +11.3pp sits inside HarnessBank's credited 9.2–15.4pp band, and both sit well inside the ~44pp scaffold-swing envelope Measuring Beyond Accuracy Saturation measures for scaffold choice on a fixed model. (Qualified 2026-08-04 by Rethinking the Evaluation of Harness Evolution for Agents: agreeing on magnitude is weak evidence when both magnitudes are measured against a do-nothing baseline. Budget-matched, plain parallel sampling buys +4.1pp without unit tests and +13.1pp with them on the same benchmark — so a gain in this band is not by itself evidence that the harness got better.)
• They agree on where the gains live. Cline's agent added retry logic, output-aware loop detection and PID tracking; HarnessBank's accepted edits span all four levers and its prompt-only baseline is credited nowhere. Agent-authored harness optimization is a runtime and control-flow activity, not prompt engineering — which is also why it grows the harness rather than shrinking it. Wang et al.'s meta agent independently reproduces the escalation (prompt rules → middleware enforcement → tool-layer fixes) and finds it insufficient, so the third source agrees about the layer while disagreeing about the payoff.
• They differ on what can be claimed. See the RSI section below: HarnessBank's cross-model dissociation is the strongest available argument that this loop cannot be recursive self-improvement in the definitional sense.
• HarnessBank's own limits. Single lab; code "publicly available upon acceptance" and unreleased at ingest; one default backbone for the main table; SWE-bench underpowered at n = 26. Most importantly the held-out tasks are held out within each benchmark — nobody has run a harness evolved on suite A against an unrelated suite B, which is a different transfer claim from the one measured here.

This is not the capital-R RSI the title claims#

Cline's post opens by invoking Recursive Self-Improvement and linking Anthropic's RSI essay, and closes with "recursive self-improvement is no longer a sci-fi experiment." The vendor's framing is a claim, not a definition, and the two senses should not be merged. What is documented here is narrow scaffold hill-climbing on a fixed benchmark:

	Cline's run	RSI as this wiki defines it
What improved	the harness around the model — retry policy, loop detector, process handling	the model itself; each generation designs and trains its successor
Where the gain lives	one repo, one benchmark, one model pairing	general capability, transferring across tasks
Who supplied the direction	a human-written brief with a pinned end state and an explicit anti-reward-hacking clause	the system, autonomously
Compounding	none demonstrated — a single campaign, not a loop that reruns on its own output	the whole point: each round makes the next round faster
Termination	17 hours, human pressing continue when it stalled, human PR review before merge	the open question the RSI literature is about

The model's weights were untouched. The improved artifact was a pull request against a TypeScript codebase. Nothing about the run bears on the extrapolation in When AI builds itself except by loose analogy — it is a data point for AI Accelerating AI Development (AI compressing AI-adjacent engineering work), which is that essay's present-tense empirical half, not its projection. The honest reading: an instance of the perspiration-becomes-automated argument, at the scale of one repo's scaffold.

There is a real, testable boundary between the two: does the improvement transfer? A harness patch that lifts held-out agentic tasks the agent never saw is a different object from one that lifts the suite it was optimizing against. Cline reports no transfer measurement.

HarnessBank runs that boundary test twice, and both results point the same way:

• Across tasks, within a domain: it transfers. Six of seven benchmarks credit the evolved harness on sealed test tasks. So harness evolution is not pure benchmark memorization — the thing being fixed is a real failure mechanism, not the training tasks' identities. (Contested 2026-08-04 by Rethinking the Evaluation of Harness Evolution for Agents. HarnessBank's finding stands on its own splits and is not withdrawn — but the same test, on Terminal-Bench 2.1 with frontier backbones and a different evolution method, returns +1.2 / +0.0 / +0.6 average, and the authors read the discrepancy against evolution's same-task results as evidence that "the revisions discovered during evolution encode task-specific shortcuts rather than genuinely better harness design principles." Within-domain transfer is now a contested result whose sign appears to depend on baseline headroom, not an established one.* Updated 2026-08-13 by DarwinX: Evolving Agent Harnesses Through Natural Selection, which supplies a third and a fourth data point pointing opposite ways within one system: WebArena-Infinity transfers hugely (300 synthetic intents → 1,260 unseen real tasks, +49.5 audit-clean) while TerminalWorld's 41-task held-out split returns three tasks at McNemar p = 0.45, with a pre-search reference matching the headline. So the sign depends on something inside a single paper's own results, which rules out lab, method and backbone as the explanation and leaves the starting harness's condition — the discriminator argued below.*)
• Across models: it does not, by construction. The evolved harness is near-zero off its matched pathology and actively harmful (-15.7) when the same lever is turned the wrong way for a different model family. What generalizes is the procedure, not the artifact. Qualified 2026-08-13, and in the direction the frame predicts. DarwinX runs a GPT-5.5-evolved TB2.1 harness unchanged on an Opus 4.8 base and on a different benchmark (SWE-bench Verified) and it holds — 84.2% official pass@1, +3.4 over an LSP fix-skill reference, no in-domain feedback. That is not a counterexample to the matching law, it is the law's own fine structure: what HarnessBank transplanted and lost was a reasoning-budget lever (thinking on/off, raise-reasoning), pure solver pathology; what DarwinX transplants is a verification / artifact-contract bundle whose seven skills contain no domain knowledge and no effort setting — "state the acceptance condition, then check against it before committing" is a task-domain regularity, so What Makes a Self-Improvement Artifact Transfer?'s general form predicts exactly this outcome. Two limits travel with it: the SWE-V arm has no matched-model base-Monet comparator, so the +3.4 is against a different harness rather than an unevolved one, and every compared harness sits inside 80.8–84.2. And DarwinX's own cross-model instability is visible elsewhere — on TerminalWorld the same procedure on GPT-5.5 reaches 56.1%, below the neutral Terminus-2's 61.0% on that base, with Opus 4.8 reported as the headline.
• And the loop converges rather than compounding. Under the significance gate it stops at its 10-round floor; the ungated variants only stop at the cap because they keep hallucinating progress. Nothing here compounds without a human re-issuing the brief for a new domain and a new model.

A process that produces a model-specific correction and then terminates is a fitting procedure, not a system designing its successor. That is the sharpest separator the corpus has for the vocabulary question above, and it comes from the strongest-evidence source in this cluster.

Evidence weighting: the vendor scores its own harness#

Four things bound how much this result can carry:

• Self-reported, not replicated. Cline ran the benchmark, on Cline's harness, and published its own scores. No third party has rerun it. Traces and cost breakdowns are posted as gists, which is more than most vendors do, but posting artifacts is not replication.
• The benchmark is the one they have been hill-climbing for months. Cline says so plainly — Jan 2026 on Opus 4.5, a Feb 2026 published playbook, this run in July. Repeated optimization against a fixed suite by the party being scored is the setup Benchmark Contamination and Decontamination and Evaluation Awareness & Grader Gaming are skeptical of, even when no verifier is touched and no task name is detected. Overfitting to a benchmark's failure distribution is a live risk that "we didn't reward hack" does not address: a retry policy tuned on the five 429s in this suite is fitted to this suite's provider conditions.
• The optimizer had write access to its own scoring substrate. See below.
• A prompt-level anti-gaming clause is not a control. The prompt banned verifier edits, task-name detection, and timeout inflation; Cline reports the guardrails held and the model policed itself (recording attribution guards, excluding two runs invalidated when the orchestrator was accidentally terminated). That is a good result and it is also self-attested.

Two runs were thrown out and rerun; the reasoning-effort fix got no causal credit; some failures were judged genuine model limitations. The post reports its own negative results, which is the main reason to take the rest seriously.

The same-model-different-scaffold effect this campaign exploits is measured independently in Measuring Beyond Accuracy Saturation — swapping the scaffold on a fixed model swings accuracy ~44pp on CORE-Bench Hard, and two scaffolds on one model disagree on 31% of tasks. An 11.3pp harness-only gain is unremarkable against that baseline; what is new is that the agent found the 11.3pp rather than a team of engineers.

Decoupling by prompt, not by architecture#

Optimizer–Evaluator Decoupling's invariant is that the thing proposing a change never grades it. This campaign violates the architectural form of the rule and substitutes a behavioral one: the optimizer had write access to the repo that runs the eval, so nothing structural prevented it from editing the grader. What stood in for the split was (a) a prompt clause forbidding verifier edits, task-name detection, and timeout inflation, and (b) a human reviewing the final PR before merge. Both are the weaker kind of guarantee — the Bun stub-and-justify episode is the corpus's evidence that a proxy metric plus write access is exactly the configuration where gaming appears, and there the patch had to be applied at the grader, which was only possible because grader and optimizer were separate.

The generalizable design note: when the harness under optimization is the eval substrate, decoupling has to be reintroduced deliberately — a frozen eval harness the agent cannot edit, a held-out suite it never sees, or a grader run from a pinned commit. Cline's campaign used none of the three; it used a human at the end. That worked here at 89 tasks and one PR, and it is precisely the check that stops scaling (Verification as the New Bottleneck).

HarnessBank deploys all three — immutable kernel, sealed test split, deterministic evaluator owning every score — and then ablates the last one, which is the corpus's first measurement of what the split is worth rather than an argument that it should exist. The measured answer is unintuitive: on TB2 the gate changes nothing about what ships, and everything about whether the loop can stop.

The complementary experiment: what happens when the gate itself is self-authored#

HarnessBank ablates a significance test off an evaluator that stays exogenous in every arm. Guo et al. (Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents, CAS, arXiv 2607.24300, empirical — full treatment on Optimizer–Evaluator Decoupling) ablate the exogeneity itself: the agent co-edits policy.py and tests.py each round, and the only thing held back is a deployment evaluation that never enters its context. The two papers bracket the same design space from opposite ends, and reading them together separates two costs that are easy to conflate:

• An exogenous evaluator without a significance gate costs you the stopping rule. HarnessBank's ungated variants ship the same TB2 score, admit two noise mechanisms as elites, and see phantom progress in 62–76% of post-convergence rounds so the loop runs to its cap.
• An endogenous evaluator costs you the artifact. Guo et al.'s unprotected runs end with self-scores of 0.70–1.00 while 15 of 35 model-game policies land below the game's random reference, and a traced Breakout run peaks at 17.6, is overwritten to 7.5, rediscovers 18.1, and finishes at 12.2 with its self-test pass rate near 1.00 throughout. Restoring only the exogenous comparison moves Gemini-3-Flash from 7.9 to 30.0 on the same benchmark.

Two further results speak directly to design choices this page records.

The endo-gate arm is Cline's configuration under laboratory control. It supplies a gate and whole-state rollback but sources its evidence from the agent's own tests — i.e. the split restored behaviorally rather than architecturally. Compute-matched it does most of the work (mean final truth 7.7 → 13.9 against SEAL's 15.4, peak-to-final loss 6.9 → 0.5 against 0.4), so the retention benefit is mostly the conservative update, not the exogeneity. Uncontrolled it is unreliable — below no-protection for three of six models and timing out on a fourth. That is the sharpest available reading of what Cline's prompt clause plus human PR review was substituting for: the cheap half, with the consistency left to chance.

Nobody has tested whether HarnessBank should also hide its numbers. Guo et al.'s leaky-anchor arm holds the audit fixed and only discloses its scores after a rejection; SEAL is at least as high in all six rows and strictly higher in five, with a 35.1 → 12.7 reversal on the worst cell. Audit confidentiality is a fifth design condition beyond HarnessBank's kernel/split/evaluator triad, and it is untested in the harness setting where the evolver reads execution trajectories by design.

The domains do not overlap — programmatic Atari policies against seven agentic and coding benchmarks — so this is convergence, not replication. But both labs land on the same asymmetry: the optimizer may reason about the metric freely, provided reasoning cannot become credit.

The human role, compressed#

The before/after Cline offers is the post's real argument:

• Jan 2026 — Opus 4.5, Terminal-Bench, 47% → 57%. Four engineers, a couple of weeks, reading traces and staring at failure logs by hand. Published as a playbook in Feb 2026.
• Jul 2026 — Kimi K3, Terminal-Bench 2.1, 77.5% → 88.8%. One prompt, one engineer, 17 hours, human intervention "nearly zero" (pressing continue every few hours on a cloud VM).

Not a controlled comparison — different model, different benchmark version, different starting harness, different absolute band on the difficulty curve. What it does show is the human's job moving from doing the loop to writing the brief and reviewing the PR: the same narrowing Harness Shrinkage as Models Improve documents for scaffolding, arriving at the meta-level. Cline's own conclusion — "the bottleneck isn't models but the humans using them" — is the practitioner form of that claim, and it is the part most likely to be true independent of the benchmark number.

Note the direction of the edit, though: the agent added harness code (retry logic, output-aware detection, PID tracking), it did not delete it. Agent-authored optimization is not automatically shrinkage; here the model was patching scaffold that was too thin, not pruning scaffold that had become drag.

Connections#

• GDPval Benchmark — the domain in HarnessBank's suite whose primary benchmark this wiki now holds, and the place where an evolved harness and the benchmark's own authors measure the same lever from opposite ends. HarnessBank credits 43.7 → 52.9 Pass@1 (+9.2) on a sealed GDPval split; OpenAI, without any harness evolution, moves GPT-5 high 38.8% → 43.1% on human expert preference with a single generic prompt that tells the agent to render its deliverables to PNG and look at them (self-inspection 15% → 97%, black-square PDF artifacts eliminated, egregious PPTX formatting errors 86% → 64%), plus best-of-N=4 with a judge and container GET access. Two independent measurements that a substantial slice of the gap to a human professional is protocol rather than capability — and a caution for this page's headroom discriminator, since GDPval's starting harness was not broken, it was merely not looking at its own output

• Inference-Time Architecture Search — the constrained ancestor. Archon (2024) searches the same kind of object with an external Bayesian optimizer over a closed, hand-specified operation vocabulary and a hand-ordered layer grammar, offline against a held-out split, rather than letting a meta-agent rewrite harness source. Wang et al.'s budget-matched negative result is on the unconstrained form only, which suggests — untested — that the constraint is where the value is

• Continuous Self-Modification Under Review — the fourth instance and the only unbriefed one: 161 days, 1,085 self-modification commits, a blocking multi-model review gate, and no objective function. It shows the loop can run for months without a human re-issuing a brief (so termination is a property of the benchmark protocol, not of self-modification), independently reproduces the maintenance-not-design verdict, and supplies zero measurement — its benchmark scores were taken on frozen seeds with self-evolution disabled

• Cline — the vendor that ran the campaign and whose harness was the object under optimization

• Agent Harness Engineering — the patterns being edited; this page is what happens when the agent, not the engineer, does the editing

• Agent Quality Flywheel — the same eval-fix loop with a human approving each fix; Google states the autonomous version as a direction it deliberately has not shipped, and this is that version running in the wild

• Optimizer–Evaluator Decoupling — Cline violates it architecturally and restores it behaviorally (prompt clause + human PR review); HarnessBank restores it architecturally (immutable kernel, deterministic evaluator, sealed test set) and then ablates it, which is where the rule stops being a principle and becomes a number; Guo et al.'s SEAL runs the mirror ablation on exogeneity itself and adds audit confidentiality as a fifth condition nothing here tests

• Deterministic Pre-Execution Gates — third-party corroboration of this page's sharpest finding, from a lab that never mentions harness evolution. Reddy et al. (arXiv 2607.07405, empirical) hand-write four runtime predicates that intercept tool calls before dispatch and get +12.4pp on τ²-bench airline, replicated on 15 disjoint seeds — the same verdict GEPA's zero credited tests deliver, that the gains in this class are runtime control-flow, not prompt-addressable, arriving from human-authored patches instead of an evolution loop. Two honest limits on reading it as corroboration: they never ran the prompting baseline (their limitation 7), so the comparison is assumed rather than measured; and their gates were written from the policy and evaluated on the same task set with replication over seeds, not tasks — the held-out-task discipline HarnessBank supplies and this does not

• Unproductive Self-Verification — the same lever, opposite sign: HarnessBank's evolver keeps discovering a verify-finalize self-check as a credited patch on backbones whose dominant pathology is premature finalization, while Anthropic's guidance for Opus 5 is to delete verification instructions because that model already over-verifies. The pathology→patch matching law predicts exactly this, and it is the cleanest cross-source support the law has

• Large-Scale Test-Time Compute — the axis this page's newest source puts harness evolution on, and loses. HarnessBank beat one test-time-scaling arm on one domain (a 16× token budget does not reproduce a targeted control-flow fix); Wang et al. run four arms at a matched K = 5 across the whole suite and find plain parallel sampling ahead of harness evolution on every model in both feedback settings. Two directly opposed data points on whether extra compute is better spent on the scaffold or on the trajectories — and now a third that declines to run the arm: DarwinX argues its extra compute is targeted rather than equal (4.3× tokens on the six tasks it newly solves, ~1.4× on the 69 both agents already solve), which is a claim about allocation and not about budget

• Orchestration Sets Token Economics — the hand-built counterpart, and the reason the negative result here is narrower than it sounds: a designed orchestration layer moves cost −41% uniformly across six models, while a machine-derived one does not clear parallel sampling. Same object, different author, opposite verdict — the distinction is drawn above

• Harness-Induced Belief Divergence — the unmeasured side effect: the middleware an evolution loop generates (output truncation, finalization gates, turn-budget interrupts) is drawn from exactly the interface classes Yi & Song show shift the agent's belief state. Harness evolution manufactures those changes as a by-product of chasing a score and nobody has looked at what they do to what the agent thinks is happening

• Reward Hacking — the failure mode the prompt banned by instruction rather than by construction

• Recursive Self-Improvement — the term the vendor invokes; the distinction between narrow scaffold hill-climbing and model-improves-its-successor is drawn above and matters more than the shared vocabulary

• AI Accelerating AI Development — the claim this run actually supports: AI compressing AI-adjacent engineering work (weeks of four engineers → 17 unattended hours)

• Harness Shrinkage as Models Improve — the human-role-narrowing dynamic reaching the harness-authoring layer, with the twist that the agent grew the harness rather than pruning it

• Open-Ended Discovery Harnesses — the same budget-matching charge applied to a different search target, with a split answer. SwarmResearch searches over solutions rather than harnesses, so a loss would mean "spend the compute elsewhere" rather than "the artifact doesn't transfer" — and its comparison is matched against the multi-agent baseline CORAL ($50/task both, same runtime, same model, and the margins there are small enough that the authors call three of the wins polishable) but not against the evolutionary baseline EvoX (~$23.50/task average against $50, and that is where the large margins are). Half a control is the current state of the art in this literature

• Client-Side Agent Optimization — the sibling lever: AgentOpt searches over model assignments holding the harness fixed; this searches over harness patches holding the model fixed. Both are client-side, and neither has been measured against the other

• Cost-per-Task Over Cost-per-Token — the campaign's economics: ~$680 and ~1B tokens to move a per-run cost from $79 to $49.8, with the payback boundary undefined

• Measuring Beyond Accuracy Saturation — the independent measurement of how much scaffold alone can move a score (~44pp swing, 31% task-level disagreement between scaffolds on one model)

• Compute-Controlled Benchmarking — three ways this page's results are uncontrolled. Cline's "88.8% at $49.8 vs Fable 5's $552" is a harness-uncontrolled comparison; the whole method class is compute-uncontrolled until someone runs a matched-budget baseline, which is Brown's benchmark-maxxing critique landing on a named method with an experiment attached (the pass@1-versus-pass@5 split is the portable diagnostic that came out of it); and DarwinX is that page's cleanest specimen of a categorical effort tier presented as a compute control — "medium / high / xhigh" defined nowhere, no compute column in its protocol appendix, no dollar figure, and its own headline pair crossing default → high

• Evaluation Awareness & Grader Gaming — the adjacent threat model: months of optimization against one suite is a validity risk distinct from, and not excluded by, the absence of verifier edits

• Verification as the New Bottleneck — the human PR review is the last decoupled check in this loop and the one that does not scale with campaign count

• Kimi (Moonshot AI) — the model under test; Cline reports matching Moonshot's own 88.3% vendor SOTA with a general-purpose harness

• Capability-Gated Model Fallback — the abandoned arm of the experiment: Cline dropped Fable 5 as leader model because the safety classifier kept downgrading it to Opus 4.8

• Knowledge-Centric Self-Improvement — the third axis and the direct contradiction on artifact portability: freeze a curated knowledge base instead of a harness and it transfers across LLM families in every cell. Both papers independently rank prompt optimization last. Wang et al.'s §5.1 supplies the mechanism that separates them — harness edits "memorize fixes rather than distilling strategies," which is precisely the selection Caltech's distillation stage exists to enforce — and the same budget-matched charge has never been run against the knowledge axis

• What Makes a Self-Improvement Artifact Transfer? — the synthesis of this page's transfer results: the matching law's fine structure (patches transfer within a pathology class, across families) generalizes to "an artifact transfers exactly as far as the regularity it encodes extends," which unifies the cross-model transplant failure here with cross-release instruction depreciation and yields the stated RSI separator

Open Questions#

• Does agent-authored harness evolution actually beat simple test-time scaling, and does it generalize to held-out tasks? Partially answered — and the two halves now point opposite ways. Test-time scaling: answered, negatively. Rethinking the Evaluation of Harness Evolution for Agents (empirical) supplies the full-suite budget-matched arm this bullet named as missing — four methods at K = 5 on Terminal-Bench 2.1 across three frontier models, with and without unit-test feedback — and harness evolution finishes below plain parallel sampling everywhere and below do-nothing on average without unit tests (67.4 vs 68.2), with its pass@5 gain traceable to multi-attempt selection rather than pass@1 capability. Held-out generalization: contested, not settled. HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution credits six of seven sealed per-domain tests at z ≥ 1.96 (+9.2 to +15.4pp); Wang et al.'s 45/10/34 split on Terminal-Bench 2.1 returns +0.6pp average and +0.0 on GPT-5.4. The proposed discriminator is baseline headroom (superseded 2026-08-13: score headroom fails on the decisive pair — Wang et al.'s 68.2 baseline is below base Monet's 75.5 and gets the smaller gain; the surviving discriminator is how broken the starting harness was.) Still missing entirely: transfer to a different benchmark, which no source measures. Settled 2026-08-13 for the cross-benchmark half by DarwinX: Evolving Agent Harnesses Through Natural Selection: a TB2.1-evolved harness runs unchanged on SWE-bench Verified and on a different base-model family for 84.2% official pass@1, +3.4 over an LSP fix-skill reference — with the caveats that there is no matched-model base arm on SWE-V and every compared harness sits in 80.8–84.2. The same source raises the held-out gain on a huge-headroom suite to +49.5 (WebArena-Infinity, 1,260 unseen real tasks) and returns p = 0.45 on a small one, so the held-out half stays contested and the budget-matched half stays unanswered — DarwinX defines no effort tier, publishes no compute column and no dollar cost.
• Is the negative result a property of harness evolution or of Terminal-Bench? Wang et al.'s §5.2 names two conditions a fair test needs — substantial headroom above current agent scores, and a benchmark whose performance genuinely depends on the harness (specialized tools, skills, workflows) rather than one where "a shell tool and a basic prompt already suffices." Terminal-Bench satisfies neither for frontier models, and it is the benchmark essentially the whole harness-evolution literature reports on, including HarnessBank and Cline. Falsifiable directly: run the same four budget-matched arms on a harness-sensitive suite and see whether the ordering inverts. Partially answered 2026-08-13 — the second condition, not the first. DarwinX: Evolving Agent Harnesses Through Natural Selection's WebArena-Infinity arm is the harness-sensitive suite by accident (a coding agent's harness pointed at Chrome), and evolution moves it 43.5% → 93.0% audit-clean on 1,260 held-out real tasks. What is still un-run is the budget-matched version of that experiment: nobody has put parallel sampling and harness evolution on the same suite at the same normalized budget where the harness genuinely binds.
• Is the Wang-et-al./DarwinX disagreement actually a search-budget disagreement? The two papers cap the search four orders of magnitude apart and neither reports it in a comparable unit: Wang et al. fix K = 5 rounds of meta-agent edits; DarwinX evolves "over many generations," screening at avg@3 on rotating subsets and confirming steering nodes at avg@5 on the full 89-task suite, and states no generation count, no rollout total and no dollar figure anywhere (the one reconstructible arm is WAI's 26 kept / 36 reverted iterations, each screened at avg@3 over 300 intents). Falsifiable two ways: cap DarwinX's generations at Wang et al.'s K and see whether the gain survives, or have both papers publish total rollouts.
• How much of the 77.5%→88.8% survives on an agentic benchmark Cline has not been hill-climbing for six months? Every fix is described as a general harness improvement, which is a transfer claim, and no transfer measurement is reported. (HarnessBank does not touch this: it measures within-domain held-out tasks, not cross-suite transfer.)
• Does the semantic quality-diversity archive earn its complexity against a plain greedy keep-the-best loop, holding the significance gate fixed? HarnessBank ablates the gate but never ablates the gene bank; its archive evidence is observational (credited harnesses stack mechanisms from more than one cell), not a controlled arm. Sharpened, not answered, 2026-08-13. DarwinX: Evolving Agent Harnesses Through Natural Selection builds its whole contribution on the archive plus a cross-lineage merge operator and still does not ablate either — §9 says so in its own words ("the contribution of recombination relative to single-lineage mutation still requires controlled ablation"), its archive evidence is the same observational shape (four specialists solve 24/25/26/27 of 41 and the merge solves 28, on a split where one task is 2.4 points), two of those four specialists fail to beat the unevolved base, and on WebArena-Infinity every merge was reverted so the largest gain in the corpus accrued along a single lineage with the recombination machinery contributing nothing. Two papers now rest on the archive and neither has isolated it.

Resolved Questions#

• Is there a stated test that separates narrow scaffold hill-climbing from the recursive self-improvement the term names — e.g. capability transfer to unrelated tasks, or improvement that compounds without a human re-issuing the brief? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? states the test: name what the improvement artifact is fitted to, then measure whether gains survive outside that scope. HarnessBank's two separators (cross-model transplant near-zero off the matched pathology and -15.7 wrong-way; termination at the 10-round floor vs phantom progress in 62–76% of ungated post-convergence rounds), Wang et al.'s compute-side separator (at matched budget the loop does not beat sampling five times), and Caltech's orthogonality result (a domain-fitted artifact compounds portably while the model improves not at all) instantiate it: solver-fitted gains are maintenance, domain-fitted gains are assets, and RSI would require the gains to live in the solver itself. The stated test's un-run instances remain evidence gaps: cross-suite transfer stays in this page's #oq/source items, the two-artifact controlled test on Knowledge-Centric Self-Improvement, and no source yet runs a loop that re-issues its own brief.

Sources#

• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Patwardhan et al. (19 authors, OpenAI), arXiv 2510.04374 v1, 2025-10-05, 29pp (empirical). Cited here only for §3.4 and Figure 9: the reasoning-effort ladder (GPT-5 low 32.7% → high 38.8%; o3 29.8% → 34.1%), the prompt-tuning arm (38.8% → 43.1% human-preference win-or-tie, with the multimodal self-inspection and formatting-error deltas), and the scaffolding changes (best-of-N=4 with a GPT-5 judge, GET requests in the container). The prompt itself is reproduced in full in Appendix A.3. Vendor-authored; full treatment and COI on GDPval Benchmark
• Sidekick's continual learning loop — Andrew McNamara & Cody Mazza-Anthony, Sidekick's continual learning loop, Shopify Engineering, 2026-08-05, case-study (first-party account of the authors' own production system; no controlled arm, no held-out split, no gain figure for the harness stage in isolation). Cited for the sixth-instance section only: the "optimization target is the entire harness" framing, the program.md autoresearch config and its editable prompts/ / tools/ / harness/ scope, the propose-evaluate-keep-or-discard loop with a separately-calibrated judge as the metric the optimizer cannot edit, and the plateau statement ("once harness improvements plateau, we begin optimizing in parameter space") — which is an assertion with no score, round count or spend attached. Full source treatment on Agent Quality Flywheel
• Recursive Self Improvement for Coding Agents — Ara Khan, Cline blog, 2026-07-24 (case-study): the five-experiment walkthrough with commit hashes, baseline/candidate/confirmation scores and costs, the Jan-2026 four-engineer contrast, the ~$680/~1B-token accounting, the anti-reward-hacking prompt clause, and the abandoned Fable 5 arm. Vendor-run benchmark on the vendor's own harness; scores self-reported, not third-party replicated
• Rethinking the Evaluation of Harness Evolution for Agents — Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi & Teng Xiao (Allen Institute for AI / University of Washington, arXiv 2607.12227, 2026-07-14, empirical, 13pp; code at github.com/rethinking-harness-evolution): the abstract's two methodological charges (matched feedback and inference budgets; search and evaluation sharing one benchmark); §3.2–3.5 the four-method budget formalism including harness scaling as the paper's own instance-guided control; §4.1 the setup (Terminal-Bench 2.1's 89 tasks, three models, K = 5, m = 1, 128k generation budget, high reasoning effort, two runs averaged); §4.2 + Table 1 the no-unit-test results; §4.3 + Table 2 the unit-test results and the pass@1-versus-pass@5 inference; §4.4 + Table 3 the 45/10/34 generalization split; §5.1 the memorize-versus-distill diagnosis and the context-bloat note; §5.2 the two conditions a fair harness-evolution benchmark must satisfy; Appendix A.1 the minimal seed harness (one bash tool), A.3 the disabled AHE explore agent, A.5 scoring infrastructure exceptions as failures; Appendix B the per-task harness-scaling case studies. Tables 1–3 verified exact against the PDF at ingest — no collapse, no shift — so this document's tables are citable; Figure 1 read per the image two-pass rule and independently reproduces Table 1's Average column (72.3 / 69.3 / 67.4 / 71.8 against a 68.2 baseline). Limits: single benchmark, two runs per cell with no variance, error bars or significance tests reported anywhere, harness evolution instantiated only as AHE with its explore agent disabled (HarnessBank was not run), and the authors' own §5.2 concedes Terminal-Bench may be a poor instrument for the question
• DarwinX: Evolving Agent Harnesses Through Natural Selection — Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang, Rishi Mullur, Thai Hoang, Zhiyuan Hu, James Zhu, Phil Mui, Silvio Savarese, Ran Xu & Zeyuan Chen (Salesforce AI Research / Salesforce Agentforce, arXiv 2608.07545, 2026-07-31, empirical, 33pp / 14 tables / 10 figures): §2 the preserve-and-extend contract, the fitness enabler (g > 0, R ≤ δ), parent selection by cumulative lineage gain, the merge operator's union-coverage acceptance rule, and the three learning signals; §3 the four-regime evaluation ladder and §3.2's own statement that "public leaderboard rows use different models and effort settings, so they provide context rather than controlled comparison"; §4 + Table 2 the TB2.1 leaderboard rows; §4.1 the self-described "effort-controlled comparison" and Figure 5's per-task compute medians; §4.2 the reward-hacking audit (370 rewarded trajectories, two flagged, one confirmed shortcut); §5 + Table 3 TerminalWorld held-out and §5.1 the 31.7-point proxy/truth gap and the McNemar caveats; §6 + Tables 4–5 WebArena-Infinity, the two-stage validity detector and the 293 → 17 invalid-trajectory drop; §7 the SWE-bench Verified transfer; §8 + Table 6 the seven-skill verification/artifact-contract bundle; §9 the limitations, including "inference effort [is] not independently randomized"; Appendix B Table 8 the per-benchmark protocol table (no compute column); Appendix C Table 9 the retry sweep and the pre-TW reference that also reaches 28/41; Appendix D.1/D.3 the synthetic-intent construction and baseline provenance; Appendix E the unmeasured harness-across-model-generations question. COI: Salesforce evaluating Salesforce's own proprietary agent (Monet) — Cline's structural position with an empirical protocol on top; the WAI validity audit is applied only to the authors' own trajectories, which they argue (correctly) makes the external comparison conservative for them. Parse verdict: clean. Tables 1–5 were spot-checked cell-for-cell against pdftotext -f <page> -layout at ingest — captions above all of them, zero collapse, zero shift; Tables 6–14 rest on the automated pass (0 collapse / 0 shift) plus canary-recall 20/20, recall 1.00, and no page was dropped. The only artifact is cosmetic decimal splatter in prose ("2. 2-6. 0-point"), not in any cell. Figures 5 and 6 read under the image two-pass rule and both carry numbers the prose omits: Figure 5's already-solved token medians (125K → 172K) against prose that says compute "barely moves," and Figure 6's specialist bars showing two of four specialists at or below the unevolved base. Limits: single lab, preprint; no effort tier defined anywhere and no dollar cost; TB2.1 reported in-domain; TerminalWorld at n = 41 with McNemar p = 0.45 (vs base) and p = 1.0 (vs Claude Code); no matched-model base arm on SWE-V; no ablation of the archive, the parent selector, the merge operator or the effort setting; the "about 17 points on average" abstract figure is a mean over four heterogeneous benchmarks with different metrics and is not a quantity this wiki cites
• Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution — Razzhigaev, Gritsaev, Kaznacheev, Dragunov, Yampolskiy & Kuznetsov (MSU / Skoltech / Joi Lab / AIRI, arXiv 2608.08311, 2026-08-08, case-study — downgraded from empirical at compile): §3 the commit pipeline and the two modes of core evolution, §4 the 161-day deployment and its two worked evolution traces, Table 4's counters, Appendix C's evolution off scaffold disclosures. Total author COI; baselines cited rather than re-run except on SWE-bench Pro, where the head-to-head is a null (58.2% vs Codex 59.4%, McNemar p = 0.40 on 655 paired tasks after a symmetric contamination filter). Parse warnings (Table 2 collapsed, Table 4 row-shifted past a clean checker verdict) and full treatment on Continuous Self-Modification Under Review
• Knowledge-Centric Self-Improvement — Wang, Yoon, Qu, Wang, Sehgal, Mazumdar & Yue (Caltech, arXiv 2607.19592, 2026-07-21, empirical): §4.4 held-out knowledge transfer (the positive-transfer counterpart to HarnessBank's §4.6 negative), §4.2 the second independent verdict on GEPA, §4.1 the cost comparison against DGM and HyperAgents. Full treatment and parse notes on Knowledge-Centric Self-Improvement
• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution — Luo, Xue, Wang, Hu & Deng (EverMind AI / Shanda Group, arXiv 2607.13683, v1 2026-07-15 / v2 2026-07-30, empirical): §3.2 the semantic gene bank and quality-biased parent selection, §3.3 Gated Harness Screening, §4.2–4.4 the seven-benchmark table and credited held-out gains, §4.3 the GEPA/DGM comparison under a matched rollout budget, §4.5 the LLM-hypothesis caveat on pathology labels, §4.6 the cross-model matching law and the 16× token-budget control, §4.7 the paired-2σ ablation. Tables 1–3 reconciled against the prose on 2026-08-03 (parse clean, no collapse or shift); Figures 3–4 read from the page images. Preprint, single lab, code not released at ingest ("publicly available upon acceptance"), and both baselines were re-run by these authors under their own protocol
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents — Guo, Cao, Yuan, Wang, Wang & Wang (Chinese Academy of Sciences, arXiv 2607.24300, 2026-07-27, empirical): Finding 1 (self-authored verification loses deployment meaning: 35/35 cells above 0.70 self-score, 15/35 below random reference), Finding 2 (the endo-gate and leaky-anchor arms, the compute-matched pilot separating rollback from exogeneity, and the Breakout trajectory trace). Parse warning: its Table 7 is collapsed and fragmented and is not quotable — cross-game claims come from Finding 3's prose (2026-09-07: since hand-rebuilt in the raw against the PDF). Full treatment and the remaining parse notes on Optimizer–Evaluator Decoupling
• How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes — McManus, Ran & Weight (Bridgewater Associates), LangChain channel, 2026-07-24, 25:44 talk, case-study. The seventh instance (15:24–18:08): agent-written failing benchmark, iterate on context repositories and harness until it passes, regression-check the suite, open a PR to Slack. No efficacy number of any kind — admitted for mechanism only. See Bridgewater Associates and Production-Sourced Evaluation