Harness or Weights: Choosing the Lever by Failure Composition

Sources#

• ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model
• Harness Evolution Hits a Ceiling: When Weight Training Should Begin

Summary#

A long-horizon agent can be improved in two places: the harness around a frozen model (prompts, memory, tools, gates) or the weights. Agent-Authored Harness Optimization collects the loops that pull the first lever, and Rationale Bootstrapping (STaR) is the ancestor of the second. Tian, Hu, Wang, Lu & Wu (arXiv 2610.11655, 2026-10-08, empirical) are the corpus's first source to pull both levers on the same models and tasks under one calibrated noise model, then cross them (seed vs evolved harness × base vs trained weights) to see which gains the trained model keeps and which still need the runtime.

Their answer is a diagnose-then-intervene rule, applied twice:

1. Read the failure composition, then pick the lever. A deterministic checker labels each failed trajectory by the first signal that fires. Process failures (harness friction, loops, exhausted step budgets) are what harness evolution removes. Content failures (a plan was delivered and it is poor) are what it leaves, and what weight training is for.
2. Read what the accepted edits changed, then decide what to train in. Harness gains that changed what the model writes (tools, prompts, lessons) trained into the weights. A gain that changed what the model sees (two pages of history instead of one) did not.

The paper's practical value is less the rule than the evidence behind it: a 48/72 development/held-out split fixed before any experiment, four-rollout means against a measured noise floor, fresh anchors after every accept, a placebo adapter, and nine evolution lines across eight models from six families.

The taxonomy: keyed to the lever, not the error#

Every trajectory whose delivered plan scores ≥ 0.5 is healthy. Every other one gets the first of four labels that fires (§3.1):

The contrast with the attribution literature on Automated Failure Attribution is deliberate (§2.1). MAST, TRAIL and the 78-type catalogues are organised by kind of error and need an LLM or human annotator. This one is organised by which lever removes the error, and it is computed by a script from process signals plus the scorer. That makes it cheap and reproducible, and it also makes it blind by construction: nothing in an F4 trajectory's process signals separates it from a healthy one, so F4 is the class no process-level fix can see.

The score identity. On DeepPlanning an undelivered plan scores zero, so composite score S = r·q̄ exactly (delivery rate × mean score among delivered plans), and ΔS = q̄·Δr + r′·Δq̄. A lever that only moves delivery is capped at q̄, the quality of what the model already writes. This is the concrete two-term case of the product decomposition on Harness Value Is a Product, Not a Score — Why the Artifact-Payoff Questions Keep Returning Partially Answered, and it explains the three null lines below in one line of algebra.

Harness evolution: real, held-out, and stopped at F4#

The loop is AHE-style (§4.2): run the harness on the 48 development tasks, build an evidence package (score, healthy rate, F1–F4 counts, process metrics, worst cases), have a proposer (Kimi K3) return two candidate edits each with its own rollback condition, judge each against the round's anchor with fixed house rules, keep the better accepted one, and re-measure a fresh 4-rollout anchor before the next round. Editable surface: prompts, policy parameters, middleware, tools. Seed: a state ledger, an exit gate, a mid-trajectory checkpoint, four hard blocks.

(Table 5 for scores, re-measured outside the loop; Tables 6–7 for composition and decomposition. The two tables are different nights, which is why the 4B delivery reads 52%→90% in one and 59%→88% in the other.)

Three things here matter beyond DeepPlanning:

• The held-out gain matches or exceeds the dev gain in both lines, on a split no loop ever saw. This is a clean positive for the held-out half of Agent-Authored Harness Optimization's contested question, on small models whose seed harness is badly broken (4B delivers half its plans). It fits that page's how-broken-was-the-starting-harness discriminator, not the frontier-tier negatives.
• The 4B quality gain is mostly selection, not better plans. Equation 2 assigns 0.077 of the +0.136 to delivery and 0.058 to higher quality among delivered plans. But a paired comparison restricted to tasks both the bare model and the evolved harness deliver moves only +0.011 dev / +0.022 held-out (Appendix F). The higher mean reflects which plans now get delivered.
• F4 is never reduced in either line (Figure 3). For 4B it grows as loops convert into delivered-but-poor plans. For 9B it ends as the largest failure class.

What the components buy (ablations, §6.1). Enforcement is mostly noise, and capability carries the gain. On 9B, removing the checkpoint, the exit gate or the four hard blocks leaves dev within noise (+0.019, +0.017, +0.017). Removing the evolved lessons file costs 0.049 dev / 0.076 held-out, and removing the ledger costs 0.088 / 0.145. Ledger and lessons are not additive: either alone sits within noise of the bare model, and both together add +0.076 / +0.092. On 4B, removing the three invented tools costs 0.071 / 0.090 and removing the lessons file 0.067 / 0.120. The one enforcement exception is the 4B exit gate: nothing on dev, but 0.077 on held-out, where delivery drops 88%→73%. Read against Harness Shrinkage as Models Improve, this is a finer cut than "prompt scaffolding shrinks": gates and blocks that police compliance were the deletable layer here, and the state and knowledge the model writes into were not.

Why the loop stops: it optimises what its evidence package shows it#

§6.2 traces the F4 ceiling to the proposer's input rather than to harness evolution as such:

• The evidence package leads with intervention process metrics, the very components the ablations show do not move the score. The F1–F4 counts come after them, and the scorer's per-check failure composition is never aggregated.
• The proposal is one tool-less call.
• The metric definitions live outside the four directories the loop may edit. That is good Optimizer–Evaluator Decoupling, and it also means the loop cannot widen its own view.
• The accepted edits target what the package shows. No edit moved the rate at which the 9B agent submits several drafts of one plan (25.5% before, 25.0% after), a failure the scorer sees and the package never surfaced.

The authors scope the claim: "The ceiling is a property of this loop's evidence package, not of harness evolution in general." Their one attempt to surface the failure composition to the proposer, on Gemma 4 12B, produced no resolvable gain. But Gemma was already the F4-dominant case the rule routes to the weights, so the test is weak.

Two loop pathologies the paper documents (Appendix G) belong to the corpus's general harness-evolution hygiene:

• Inert accepts. An edit that never executes reads as neutral and can pass the judge. The 4B round-3 policy activated a self-check whose prompt file did not exist, another accepted edit silently removed the seed's twenty-call brake, and three WebArena-Lite middleware edits never ran because of a round-indexing bug. The fix: before an edit is judged, require evidence that its code path fired at least once. It is the evolver-side twin of the activation gate on Harness Activation and Adherence. On 9B the same gate shows from the consumer side: every tool the proposer wrote was rejected because the agent never called it, including a hotel-location validator whose own reading (0.399) was the highest of any 9B candidate (Table 4, recovered from pdftotext -layout; docling welded its rows).
• Selection optimism and ratchet slippage. Proposition 1 bounds the expected inflation of a best-of-K reading at s·√(2 ln K). With 10–30 candidates at one rollout per reading that is 0.05–0.06, above the 0.035 resolution line. Measured: 4B candidate readings of 0.316/0.327/0.323 against fresh anchors of 0.282/0.288/0.305. An earlier, unreported 9B line locked its anchor on one lucky evaluation and then rejected eight consecutive candidates. Fresh anchors after every accept are the mitigation. Three early single-rollout lines were re-measured under the full protocol before being reported, and one re-read came in at 0.44–0.46 against its single-rollout 0.535.

Weight training: the harness gain trains in, and the placebo shows it is the content#

Training data are the model's own trajectories under the evolved harness on the 48 dev tasks: the top three per task above a score threshold, giving 82 trajectories over 33 tasks (4B) and 98 over 40 (9B). Each step is one sample, with loss on the assistant output only. LoRA r = 32, α = 64, on attention (qkvo) or attention + MLP, taken at the end of epoch 1 because epoch 2 memorises. This is the STaR recipe (filter the model's own successful outputs, fine-tune, no RL), and the paper says so (§2.3).

Table 1 (same machine and night as the base cells, except the evolved-harness held-out base cells, which are closing anchors from two nights earlier), reconciled against pdftotext -layout:

Four findings (§5.2):

1. The adapter transfers. On the seed harness the better adapter lifts held-out by +0.126 (4B) and +0.131 (9B). That matches or beats what harness evolution delivered, with the harness removed. Seven of eight adapter-on-seed cells are positive.
2. Stacking on 4B, substitution on 9B. On the evolved harness the 4B adapter adds a further +0.041 dev / +0.082 held-out. The held-out figure is read against a 2-rollout cross-night base cell, so the line is 0.05. Both 9B adapters add nothing resolvable (+0.010 / +0.007). Either lever alone reaches 0.44–0.45 held-out on 9B. Appendix E finds the mechanism: on the evolved harness the 9B adapter still cuts F4 (22%→6% dev), but the freed share reappears as F3 and F1, so the adapter and the lessons file are moving the same failure class.
3. It is the content, not the adapter. A placebo trained with the identical recipe on the same trajectories, with assistant outputs permuted across samples, scores 0.158 below base on dev and 0.227 below on held-out (9B, Table 10). Its trajectories run 52–65 turns against 19. Ranks 16, 32 and 64 all gain. Full-parameter fine-tuning on a larger second set (≈200 trajectories) beats LoRA by +0.044 (4B) and +0.042 (9B) held-out.
4. Each adapter moves the class the rule assigns it (Table 9). The 4B adapter reproduces the harness's behavioural effect: delivery 52%→78% dev, 55%→78% held-out, loops down 18–19 points, F4 unchanged. The 9B adapter, on a model whose delivery is saturated, moves the content class: F4 23%→7% dev and 28%→5% held-out, with the freed trajectories becoming healthy (34%→46%, 28%→53%). Training reached the class the loop did not.

The placement result has a stated mechanism. Attention-only wins on 4B, whose "textbook" mostly teaches a behaviour (deliver). Attention+MLP wins on 9B, whose textbook teaches plan content, and the MLP blocks are where factual associations are held (Geva et al., 2021). A richer 4B textbook flips 4B to MLP as well (Table 8: qkvo +0.062, qkvo+MLP +0.117 held-out).

Against the expert-imitation collapse#

This is the result to set beside the Salesforce instance on Agent-Authored Harness Optimization (Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails, Yu et al., whom this paper cites as finding that imitation under an evolved harness "can regress"). There, LoRA-SFT of a weak model on a stronger expert's trajectories under an evolved harness regressed every task (−14.9 average) by breaking the fit between the model's planning cadence and the harness. Here, LoRA and full-parameter fine-tuning on the model's own filtered trajectories under an evolved harness gain and stack. The two are consistent, not contradictory: Salesforce's fix was on-policy correction, and self-generated trajectories are on-policy by construction. Together they narrow the collapse to off-policy imitation, with both sources on Qwen-family models at 30B and below, and neither tests RL.

When a harness gain will not train in: WebArena-Lite#

The same loop on WebArena-Lite (Qwen3.5-9B, 48 dev / 117 test, σ = 0.045 so the resolution line is 0.064) moved the dev anchor 0.234→0.375 over 13 rounds. One round-10 interface edit, keeping the two most recent full pages in context instead of one, accounts for +0.12 of it. On the 117 unseen tasks the evolved harness scores 0.256→0.344 (+0.088). Delivery rises 49%→77%, while accuracy among delivered answers stays flat (0.47→0.44): the harness carried the agent further into each task and did not make it more correct.

LoRA and full-parameter adapters trained on 157 full-score closing-harness trajectories landed within the line of base under both harnesses: 0.327 / 0.331 against 0.344 evolved, and 0.216 / 0.231 against 0.256 seed. Accuracy among delivered answers fell to 0.39 / 0.37. The authors' reading: the gain lived in what the model sees, and an adapter that is never shown the second page cannot reproduce what the model did with it. The design consequence is that whether a harness gain is distillable is knowable before any training, from the surface the accepted edits touched. Caveat: this is one benchmark, effectively one load-bearing edit, and adapters measured the night after their base cells.

This sharpens Harness Shrinkage as Models Improve's migration claim ("anything the harness teaches the model how to do via a prompt section is a candidate for migration"). Instruction-shaped scaffolding migrated here. Observation-shaped scaffolding (what enters the context window) did not, and there is no reason it would, since it is information rather than behaviour.

Where the harness lever pays: nine lines, three groups#

Section 5.4 (Figure 4, Tables 7 and 14) orders the nine DeepPlanning lines by the quality of plans the seed harness delivers. The seed measurements, all available before a round is run, sort them:

• Seed quality ≥ 0.27 and F4 not yet dominant (five lines: Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-35B-A3B thinking off and on, Tencent hy3): the harness paid +0.063 to +0.136 dev in four lines. In the fifth it paid +0.053, within that line's own noise (Qwen3.6 thinking-on, per-rollout σ up to 0.07). In the three re-measured early lines the gain is all delivery, and accuracy among delivered plans does not move.
• Seed quality 0.13–0.17 (MiniCPM5-2B, Spark-X2.5-4B, Ministral-3-8B): nothing, each within noise after five rounds. The harness still moved its process class (MiniCPM delivery 61%→71%; Spark F1 21%→7%), but the freed trajectories landed in F4, and on MiniCPM quality among delivered fell 0.173→0.156, so the product did not move. This is the S = r·q̄ cap in action: only a model that can write a plan worth scoring can cash the move. On all three, the proposer's edits went to the enforcement layer.
• Delivery saturated, F4 dominant (Gemma 4 12B, F4 40%): +0.019 dev / +0.012 held-out, within noise, which is what the rule predicts. Training on that line's trajectories then bought dev gains that did not carry to held-out, and stacking on the evolved harness added nothing (LoRA −0.042 dev, at the edge of the line). The paper leaves the weights branch "open" on this model rather than claiming it.

Two frontier API models run by a co-author (Appendix J, unmatched nights, no noise estimate, older labelling scheme) point the same way. Kimi K3 under an evolved harness gains +0.058 / +0.049, mostly as quality among delivered plans (the 9B shape), and mean steps fall by more than half. GLM 5.3 has no bare control.

Read the grouping as a retrospective sort, not a prediction. All nine lines were sorted after the fact. The thresholds (0.18, "F4 dominant") were drawn from the same lines they classify, and the three non-Qwen lines ran with the pre-extension evidence package. No line was assigned prospectively.

The decision rule, as stated (§7)#

1. Delivered plans score below about 0.18: the harness lever paid nothing on all three such models. (They were not trained.)
2. Delivery well below saturation, or F3 dominant: evolve the harness. Its gains transferred to held-out in both Qwen lines.
3. Delivery saturated and F4 dominant: train the weights. On 9B the adapter alone matched the five-round evolution line on held-out.
4. F4 residual after both: the loop as run cannot see it; expose the scorer's failure composition to the proposer.
5. A harness gain in hand: read what the accepted edits changed. Edits to what the model does trained in; a gain from what it sees stayed in the harness.

This is the first measured version of the ordering Shopify asserted without numbers on Knowledge-Centric Self-Improvement ("once harness improvements plateau, we begin optimizing in parameter space"). It adds the conditions under which the order is right: the harness stage plateaus at the content class, the weight stage is what reaches that class, and on a model that already delivers, the weight stage can make the harness stage redundant.

A vendor counter-framing (October 2026). IBM's ALTK-Evolve (vendor-claim; full treatment on Read-Time Memory Curation (JitMem)) argues the opposite default: "fine-tuning a model on a benchmark is expensive, brittle, and doesn't transfer", so learn rules into the prompt instead. Its artifact is this paper's lessons file on its own: rules distilled from the model's own train-task trajectories, with no ledger, tools or gates. It reports held-out τ²-bench gains with gpt-4.1 and gpt-5.5. That fits this page rather than refuting it. The lessons file was one of the two load-bearing components here, and IBM never runs a weight arm on the same model. Its only contact with the weight lever is a leaderboard comparison against a different, fine-tuned base model. The rule here would ask which failure class the rules removed. IBM's example rules (confirm before writes, one action per turn, parse the name before lookup) are process-shaped, which is the harness side of the split.

Evidence weighting#

• Strong for its size. A pre-registered split, a calibrated noise floor (per-rollout σ median 0.023 over 26 same-night cells, worst 0.042; resolvable difference 0.035 at four rollouts), same-night comparisons with cross-night exceptions marked, fresh anchors, a placebo, a rank ablation, ablations on both sizes, and a second benchmark. Most harness-evolution sources in the corpus report fewer controls.
• Small. 48 dev / 72 held-out tasks on one planning benchmark, models ≤ 35B, and LoRA trained on 82–98 trajectories. The 4B ablations use two rollouts (line 0.05), and the placebo exists for 9B only.
• No budget-matched baseline. The paper concedes it does not compare the loop to matched-budget test-time search (Wang et al., 2026b), the arm whose absence is the standing objection on Agent-Authored Harness Optimization.
• A known scorer bias. DeepPlanning's plan-to-JSON conversion is an LLM call that silently picks one draft from multi-draft answers (four of five converters did this in 20 of 20 cases). It inflates scores by ~0.025 on average and 0.09 on the five hardest answers. Every cell uses the same converter, but the multi-draft rate differs by configuration (25.0% for the 9B closing harness, 1.0% for 4B), so differences near 0.025 deserve care. The authors reported the bug upstream with a prompt fix.
• COI: none apparent. Two independent researchers plus Purdue, Berkeley and Stanford authors. No author ships a model in the evaluated set, and the proposer (Kimi K3) is third-party.

Connections#

• Agent-Authored Harness Optimization — the harness lever's home page and its nine instances. This source supplies a noise-calibrated held-out positive (+0.133 / +0.119 on 72 unseen tasks; +0.088 on 117 WebArena-Lite tasks) for that page's contested held-out question, two loop pathologies (inert accepts, ratchet slippage) and the F4 ceiling. It also supplies the on-policy counterpart to the Salesforce expert-imitation collapse
• Rationale Bootstrapping (STaR) — the weight lever's recipe here is STaR without rationalization: filter the model's own successful trajectories, fine-tune, no RL. The new element is where the trajectories come from. An evolved harness is the generator that lifts the model's own success rate high enough to produce a training set, so harness evolution works as STaR's data engine
• Knowledge-Centric Self-Improvement — the what-should-persist taxonomy whose weights row was a single case-study. This is the controlled harness × weights corner of that page's three-way experiment, still missing the distilled-knowledge-bundle arm
• Harness Activation and Adherence — inert accepts are the activation gate on the evolver side (an edit whose code path never fired), and the 9B line's never-called tools are the same gate on the consumer side; the required "code path fired" check is a beacon in all but name
• Automated Failure Attribution — the error-kind taxonomies (MAST, TRAIL) and LLM attributors that page measures. This taxonomy instead keys on the lever, is computed by a deterministic checker, and accepts blindness to content failures as the price
• Agent Harness Engineering — the ablations rank the classic harness components by measured value: state ledger and lessons file load-bearing and non-additive, invented tools load-bearing on the weaker model, exit gate / checkpoint / hard blocks within noise except where they hold delivery
• Optimizer–Evaluator Decoupling — the loop's metric keys live outside its editable surface, which is sound decoupling, and this is the failure mode on the other side of that wall: an optimizer that cannot edit its evaluator also cannot see what the evaluator sees unless someone puts it in the package
• Harness Shrinkage as Models Improve — the 9B substitution measures capability migrating inward from harness to weights, and the WebArena-Lite null marks the boundary: observation-shaped scaffolding does not migrate
• Recursive Self-Improvement — a harness loop that produces a model's next training set is the shape RSI arguments assume. Here it is one round, human-scheduled, on ≤ 35B models, and it plateaus at the class the proposer cannot see
• Read-Time Memory Curation (JitMem) — where the prompt-only version of this page's lessons file is treated: ALTK-Evolve's train-mined rule list on τ²-bench, set against JitMem's read-time curation, and the vendor's no-weights framing that this page's harness-then-weights rule answers
• Harness Value Is a Product, Not a Score — Why the Artifact-Payoff Questions Keep Returning Partially Answered — S = r·q̄ is a two-term instance of that page's product, measured directly on every cell rather than inferred, and it explains the null lines arithmetically
• What Makes a Self-Improvement Artifact Transfer? — the adapter is a weight-space artifact that transfers across harnesses (seed ↔ evolved) and to held-out tasks on one model, which that synthesis predicted would be the least portable artifact class; cross-model transfer of the adapter is unmeasured

Open Questions#

• Does surfacing the scorer's aggregated failure composition to the proposer let harness evolution reduce F4 on a model that writes plans worth scoring? The one attempt ran on Gemma 4 12B, the F4-dominant case the rule already routes to the weights. Settler: the extended evidence package re-run on a Qwen3.5-9B-like line (delivery saturated, quality ≥ 0.27, F4 largest class), reporting the F4 share.
• Does "edits to what the model writes train in, edits to what it sees stay in the harness" hold beyond WebArena-Lite's single context-window edit? Settler: a second benchmark where the accepted edits are observation-side (retrieval scope, history window, page rendering), with adapters trained with and without the extra observation in the training context.

Sources#

• ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model — IBM Research, Hugging Face blog, 2026-10-20, vendor-claim. Cited only for the counter-framing paragraph (the no-weights argument and the example rules). Full treatment on Read-Time Memory Curation (JitMem)
• Harness Evolution Hits a Ceiling: When Weight Training Should Begin — Yuan Tian & Bing Hu (equal contribution; independent researchers), Binghang Lu (Purdue), Hao Wang (UC Berkeley), Fang Wu (Stanford), Harness Evolution Hits a Ceiling: When Weight Training Should Begin, arXiv 2610.11655, 2026-10-08, 21pp, empirical. Cited for §3's taxonomy and identities, §4's protocol, §5.1–5.4, §6, §7 and Appendices A–J. COI: none apparent. Parse warning: PDF-derived (docling 2.126.0 / docling-mlx 0.1.1). Ingest flagged a soft table-collapse on 4 cells: the slash-paired Delivery and Mean-steps cells of Table 15 (by design, D / H pairs, confirmed by Appendix J's prose) and a welded Table 4 (9B candidate log), which was recovered from pdftotext -layout and is cited only for the never-called validator's 0.399 reading. Table 1 was verified cell-for-cell against pdftotext -layout. Tables 5–14 were reconciled arithmetically against the prose deltas they back (every delta quoted above reproduces from its table). Table 3's "0.305 / 0.269 †" is the documented context-window re-anchor, not a collapse. Figures were not re-read; nothing cited depends on figure-only content. Internal inconsistency: the abstract's 4B "held-out delivery 55%→90%" is Table 5, while the introduction's "59%→88%" is dev from Table 7 on a different night. Both are quoted with their table.