What Makes a Self-Improvement Artifact Transfer?

The question#

From Knowledge-Centric Self-Improvement's backlog (#oq/now): What distinguishes a self-improvement artifact that transfers from one that does not? The corpus holds one strong positive — Caltech's frozen knowledge bundle, positive in all eight donor-recipient cells across LLM families — and one strong negative — HarnessBank's evolved harness, near-zero off its matched pathology and -15.7pp with the lever turned the wrong way. Both pages propose fitted-to-the-task-domain vs fitted-to-the-model as the discriminator and call it "a hypothesis with two strong one-sided data points." This synthesis tests the hypothesis against the rest of the corpus.

The answer#

An artifact transfers exactly as far as the regularity it encodes extends. The task-vs-model binary the two papers propose is the special case they happened to measure; the corpus supports the general form:

• A harness patch encodes a solver regularity — one model's dominant failure mode. It transfers precisely to the set of solvers sharing that pathology and no further. HarnessBank's own matching law shows both halves: the Qwen-27B thinking-runaway stack transplants to the 397B almost loss-free (+11.0 vs +11.7 native) because the two generations share the pathology, and Gemini 3 Flash — a different family with the same careless-finalization pathology as the 397B — takes the submit-verify patch at +13.5 (z=4.15). The failure is not "crossing model families"; it is leaving the pathology class (Agent-Authored Harness Optimization, empirical).
• A distilled insight encodes a task-domain regularity — an ARC grid invariant, Rust's loop-based propagation semantics, a test-runner flag. Reusing it holds the domain fixed while the solver varies, so it survives solver churn by construction: every one of Caltech's eight donor-recipient cells improves, in both cross-family directions (Polyglot 8.3%→20.0%, ARC-AGI-1 23.3%→43.3% for the strongest pairings) (Knowledge-Centric Self-Improvement, empirical).

Stated as the design rule both pages gesture at, sharpened: know what your artifact is fitted to, because that is where it stops working. "Fit it to the problem, not the solver" follows as the practical corollary — the problem is the thing you keep when you swap models.

Four corpus tests the discriminator survives#

1. Cross-release depreciation is the same phenomenon as cross-model transplant failure#

The temporal evidence and the cross-sectional evidence unify. A verification instruction is a solver-fitted patch for a model that under-verifies; when the solver-side regularity changes — Opus 5 ships, natively at or past the right verification level — the same line inverts from helpful to harmful, and the vendor's fix is deletion, not rewording (Instruction Compounding, vendor-claim, corroborated by the system card's empirical effort-inversion). Unproductive Self-Verification draws the porting consequence explicitly: "verification steps are the most model-specific thing in [a harness], and inherited 'best practice' verification scaffolding is exactly the kind of setting that arrives with the wrong sign." That is HarnessBank's matching law observed diachronically: transplanting a patch to a different model and keeping a patch while the same model improves are both transfers across a solver-regularity boundary, and both fail the same way — the lever gets turned the wrong way for the pathology actually present.

This also closes the loop with the sibling synthesis What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful?, which sorted scaffolding by what it encodes: behavior requests (task priors about what the solver needs to be told) migrate inward and expire; boundaries, org records, identity, and serving structure survive. Its survivor classes are exactly the domain-fitted ones — a repo convention is fitted to the org, a test suite to the task's correctness conditions — and its dissolving class is exactly the solver-fitted one. One sort, two axes: what an artifact is fitted to predicts both whether it outlives the current model and whether it transplants to a different one.

2. The layer that transfers across vendors is the domain-fitted layer#

Agent Context Files — CLAUDE.md / AGENTS.md / SKILL.md — encode repo conventions, workflows, and domain procedure, and they are the one harness layer that has become a cross-vendor convention, with Genkit shipping SKILL.md loading as a second vendor's runtime in July 2026. Harness Build-vs-Buy gives the depreciation ledger for the opposite choice: a forked harness — infrastructure fitted to the moving solver frontier — falls ~4,600 merged PRs (≈13/day) behind upstream in a year, and the guidance that falls out is to customize at the highest layer that works (prompts/config → MCP → skills → SDK). Read through this synthesis's lens, that advice is: own the domain-fitted layer, rent the solver-fitted one — put your persistent investment where the regularity is yours (your repo, your workflows), and let the vendor chase the regularity that shifts under everyone (the model).

Cline's five fixes are the instructive edge case: retry-on-429, PID tracking, output-aware loop detection are fitted to neither task nor model but to infrastructure reality (provider rate limits, Unix process semantics) — the frame predicts those survive both solver churn and task churn, which is exactly Cline's "general harness bug, not benchmark-shaped special case" argument. What was fitted to the benchmark's failure distribution was their score value, and that part no one has transfer-tested (Agent-Authored Harness Optimization, case-study).

3. Transferability can be selected for at write time#

The discriminator is not just a post-hoc classifier — Caltech's protocol is a machine for enforcing domain-fitting before anything persists: concrete_primitive must be verbatim from a task (a named API, error type, flag, invariant, with a grounding quote); anti_meta_self_check rejects solver-generic tutorial advice ("separation of concerns," "two-phase pipeline"); every surviving insight carries applies_when / does_not_apply_when scoping; and the transfer adapter's final rule — if the current task conflicts with prior knowledge, prefer the task — is a supersession policy that keeps the domain, not the artifact, authoritative. Wang et al.'s diagnosis of why harness evolution fails — edits "memorize fixes rather than distilling strategies" — names the exact selection Caltech's distillation stage enforces. So the two poles differ not only in what their artifacts are fitted to but in whether the loop filters for the transferable kind: harness evolution has no gate that rejects a solver-fitted memorization, and knowledge curation is mostly such gates.

LLM-as-Compiler Knowledge Base — this vault — is the same bet made for a human-and-LLM reader: compile facts about sources (domain-fitted, model-agnostic), never prompts for a particular model, which is why the artifact is expected to survive model upgrades. Caltech is that architecture's first external empirical corroboration, now with a mechanism for why it transfers.

4. Where the artifact doesn't transfer, the procedure does#

HarnessBank's own conclusion — "a credited harness is a correction fitted to the model; the transferable asset is the diagnose→search→verify process" — recurs at the orchestration layer. Client-Side Agent Optimization's model-per-role combination is solver-fitted by construction (fitted to the current model menu's capabilities and prices), and the corpus already shows the assignment not surviving while the rule does: HotpotQA's weak-planner/strong-solver optimum inverts on Cursor's long-horizon task ($411 worker fleet under an Opus 4.8 planner vs $9,373 for frontier-does-both), and what reconciles the two is a portable principle — foreclose execution at the planner role — not any reusable assignment. The frame's prediction for its open release-cadence question: expect frontier re-runs each model release (the combo is solver-fitted), and expect the assignment rules extracted from those runs to be the durable residue.

What this does to the RSI boundary#

Transfer was already the sharpest separator between scaffold hill-climbing and Recursive Self-Improvement; the frame decomposes it into a stated test: ask what the improvement artifact is fitted to, then measure whether gains survive outside that scope. A fitting procedure produces solver-fitted corrections and terminates (HarnessBank converges at its 10-round floor; a human re-issues each brief). What compounds portably is domain knowledge — and Knowledge-Centric Self-Improvement shows compounding and RSI are orthogonal: ten unattended generations accumulate a genuinely portable asset while the model improves not at all. So "self-improvement" claims sort into maintenance (solver-fitted gains that die with the model) and assets (domain-fitted gains that outlive it), and neither is capital-R RSI, which would require the gains to live in the solver itself.

What stays open#

• The controlled test still does not exist. No one has run both artifact classes under one protocol — same tasks, same budget, same held-out and cross-model splits. The frame predicts harness-patch gains vanish off-pathology while bundle gains persist; that is falsifiable and unrun (now a #oq/source bullet on Knowledge-Centric Self-Improvement).
• The budget-matching charge is unanswered on the knowledge axis. Wang et al.'s critique — a self-improvement loop must beat the same budget spent on trajectories — has been run only against harness evolution, where it wins (+0.6pp held-out; below parallel sampling everywhere). Caltech's transfer result is immune (zero-shot single-attempt recipients), but its headline solve rates are not budget-matched against repeated plain attempts.
• Environment-fitted artifacts are predicted durable and never transfer-tested. Cline's infrastructure fixes should survive both model and task churn by this frame; no one has measured it.
• Evidence weight. Both poles are single-lab empirical preprints; the four corpus tests above draw on vendor-claim (prompting guide, Genkit, build-vs-buy) and case-study (Cline, Cursor) sources. The discriminator is consistent with everything in the corpus and directly measured by two papers — a well-corroborated hypothesis, not a law.

Design rules (compressed)#

1. Before persisting an artifact, name what it is fitted to — task domain, solver pathology, environment, org. That names where it will stop working, on both the transplant axis and the time axis.
2. Fit persistent artifacts to the problem if you want them to outlive models — and enforce it at write time. Schemas that demand verbatim task-grounded primitives and applies-when scoping (Caltech), not intentions.
3. Treat solver-fitted artifacts as depreciating stock: re-audit on every release with the ablation / dose-response detector (What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful?), and never port one to a model whose pathology you haven't checked — the sign can flip.
4. When an artifact won't transfer, extract the rule that produced it. Procedures (diagnose→search→verify, foreclose-execution-at-the-planner) are the durable residue of solver-fitted work.