Skill Revision Effects

Sources#

• Agent Skill Evolution: How Revisions Affect Coding Agents

Summary#

Every other skill-benefit measurement in this vault compares a skill against its absence (Skill Lift, Khatri's context-file ablation on Agent Context Files) or a late version against an early one on tasks that do not need any particular edit (Human-Governed Skill Maintenance). Wang, Zhao, Li, Chen, Chen, Peng & He (University of Arizona and five other institutions, arXiv 2610.04832, 2026-10-04, 21 pp, empirical) measure the thing a maintainer actually does: one revision, old body against new body, on a request built so the revised rule applies, with the Skill's name and description present in every arm. The estimand is new-minus-old on the same probe and model.

Five findings carry:

1. Skills change together with their harness. Commits that revise a Skill change harness configuration more often than size-matched commits in the same repository (23.0% vs 18.5%, +4.5 pts [+1.6, +7.3]), the excess coming from CLAUDE.md (+2.1), AGENTS.md (+2.0), other agents' config (+1.9) and Claude Code agent and command definitions (+1.4 each). Creating a Skill is coupled more tightly (+14.8 pts).
2. An added rule changes behaviour, at every level measured. Single-answer compliance +0.41 averaged over 16 open models (every one of 21 models gains); sandbox required action +0.23 over four agents (+0.16 to +0.36); blind-human-judged final correctness +0.10 [+0.04, +0.17] over the three judged agents.
3. The gain is new information. Rules naming a token the old Skill never mentioned lift compliance +0.51 to +0.66 from a 0.05 baseline; tokens already present lift it +0.08. Stating the rule adds +0.15 beyond a sentence that only names the token, and unrelated text of the same length does nothing.
4. Delivery decides how much survives. With the body loaded on demand, as real harnesses do, the four agents keep ~51% of the action gain (three open models ~38%; Sonnet 4.5 keeps all of it). At 40k tokens of agent-like context about two-fifths of the short-answer effect survives.
5. The revision is cheap; the body is not. A revision adds 18–19% input tokens to one answer but no detectable cost per agent episode (−2.3 to +3.0%, an increase above ~12% excluded), while carrying the Skill body at all costs +37 to +57% tokens per episode. One rule-model pair in five adds tokens with no detectable gain.

Evidence note. empirical, kept. Repository-clustered bootstrap CIs (4,000 resamples), Holm correction, pre-fixed equivalence margins where nulls are claimed, a 240-response blind two-annotator grader validation (κ 0.86, string-check precision 0.97–0.98, recall 0.73–0.82), and blind human judgement of sandbox artifacts (κ 0.96 / 0.90) as the primary outcome evidence. No annotator is an author. Scope limits the authors state: (1) only mechanically checkable rules that name a command, path or identifier — the literal filter keeps 17% of candidate directives and under-represents prohibitions, conditional rules and permission guardrails; (2) every effect is conditional on a request that needs the rule; (3) open models are 3.8B–32B, the closed models are three Sonnets, Haiku 4.5 and GPT-5.5, and every model runs without reasoning; (4) the sandbox is the unmodified mini-SWE-agent scaffold, which shows only the first and last 2,000 characters of a command's output, not a production harness; (5) the main corpus has no repository above 375 stars (a separate 146-repository high-star set repeats the core results). Probes are generated by Qwen 3.5-27B and Qwen models gain slightly more on them (+0.04); a Phi-4-generated probe set shrinks that to +0.007. COI: none — academic authors, no evaluated product is theirs.

How the measurement is built#

From SkillMD-138K (138,133 SKILL.md files, no history) the authors draw 1,500 repositories, rebuild every Skill's history with git log --follow, and keep the 3,159 Skills (in 856 repositories) with at least two versions — 40% of sampled files, a lower bound. 2,608 first/last pairs differ in text; a pair spans a median of two revision commits.

The unit of RQ2/RQ3 is a directive: an added or deleted line that states an imperative and names a backticked token. An LLM writes a realistic user request that needs the directive without mentioning it, and a string check grades the answer (must contain / must not contain the token, IFEval-style). 396 added-directive probes (200 repositories), 245 deleted-directive probes and 1,349 "dilution" probes around an unchanged rule. Each probe runs under three conditions — none (name and description only), old body, new body — which is exactly how a harness with progressive disclosure presents a Skill.

Effectiveness is measured at three levels: compliance in one answer (string check; includes information exposure), required action of an executing agent in a network-less sandbox (80 tasks from 57 repositories, the command actually run with its target present, the file actually read or written), and final correctness of the artifact (automated for 36 tasks; blind human judgement on all 80).

RQ1 — what revisions are, and what moves with them#

• Most revisions change a rule or procedure. 55.0% of the 2,608 pairs add or delete a MUST/NEVER constraint, conditional rule or numbered step; 19% change only examples or code; 14% only prose. The classifier's recall on MUST/NEVER and conditional lines is 0.18–0.19, so 55% is if anything low.
• Skills grow by words, not by executable surface. 60% of revised Skills end longer (median net +18 words, mean +101). Paths referenced rise 1.24 → 1.68 per Skill and shell commands 0.51 → 0.76 (Table 2, verified against the PDF), but the count per 1,000 words shows no detectable change, so Skills get longer without getting more operational. The user-invocable field rises 2.2% → 5.9%.
• Cadence. Consecutive versions are a median of under three days apart, in bursts; 66% of revision commits land more than a week after creation.
• Motive is rarely documented. 0.4% of commit subjects name a model version and 4.2% of 120 hand-labelled commits target a model or platform. Most commits add content (28%), follow a code or tooling change (20%), or fix an error (18%).
• AI authorship matches the repository's base rate. An AI-agent or bot signature appears on 37.6% of revision commits against 40.5% of other commits in the same repositories and period.
• Harness co-change is finding 1 above. The raw gap (29% vs 10%) is mostly commit size; within repository and files-changed band it is +4.5 points. Settings, hooks and MCP configuration do not co-change more than baseline — the coupling is with the instruction files, not the mechanism files.

RQ2 — what an added rule does#

Single answers. All 16 open models (six families, 2023-12 to 2026-02 releases) follow the added rule more often, by +0.34 to +0.45 (Table 3); the five closed models gain +0.42 to +0.45 on a 147-probe subset; the high-star set gives +0.36 to +0.54. The gain does not vary detectably with commit intent, executable-reference change or a same-commit harness change (smallest Holm p = 0.24). Annotators re-weighted to all main-set answers see a larger directive-following gain (+0.52) than the string check (+0.41), because the check misses more compliant answers under the new version.

Controls (four models, 292 probes). Adding only the rule line to the old Skill gives about two-thirds of the full effect; removing it from the new Skill loses ~0.21; an unchanged rule line removed elsewhere loses a similar 0.24; unrelated text of equal length has no detectable effect. Stating the rule beats naming its token by +0.15 [+0.12, +0.19], but only +0.04 where the old Skill already named the token. Prose-only or examples-only revisions move compliance with an existing rule by +0.017, inside a pre-fixed 0.08 equivalence margin, while the body itself adds +0.57 over none.

New tokens carry it. Split by whether the checked token already appeared in the old version: new +0.51 to +0.66 from a 0.05 baseline, present +0.08 from 0.61 (107 probes; partly headroom). Of 30 sampled "present" cases, 18 merely restate a rule (moved from a table to a list) and 7 promote a token from an example to a rule — and those 7 show no detectable gain. A new rule also leaks onto requests that do not need it, but faintly: +0.02 to +0.06 token use on 260 such requests, against +0.38 to +0.45 where needed.

Deletion is followed too. Removing a rule lowers compliance with it by 0.27–0.33 on eight models — about three-quarters of what adding one gains. Exploratory split by hand-checked reason: deleting a duplicate costs −0.12 (n = 27), deleting a replaced rule −0.58 (n = 24).

Generations gain alike. Twelve within-family older-vs-newer comparisons show no significant interaction after Holm correction; the data-derived bound is 0.077 (about a fifth of the main effect), 0.062 along the Sonnet 3.5 → 3.7 → 4.5 ladder. The one significant step (Qwen 2.5-7B → 3-8B on new-token probes) has the older model gaining less. The paper's maintainer advice follows: size a Skill for the smallest model that will read it, since release year matters less.

Window occupancy erodes it. The gain falls from 0.424 at low Skill occupancy to 0.270 when the Skill fills three-quarters of the usable context. Inside a coding-agent-shaped context (system prompt, 14 tool schemas, a prior exploration exchange) about three-fifths survives at 15k tokens and two-fifths at 40k; where that context already shows the token but the old Skill lacks it, the revision still adds +0.31.

Agents (Tables 4–5, verified against the PDF). A read-only tool agent gains +0.18 to +0.33. In the mini-SWE-agent sandbox, required action rises for every agent:

The Sonnet 4.5 outcome is underpowered, not null: 72 pairs give ~0.5 power for a Qwen-sized effect (MDE 0.18; ~125 pairs needed). Final correctness rises by less than the action rate on the 36 checked tasks, though not significantly.

Two confounds are tested and neither explains the gain. Stale references: the sandbox sits at the new version's commit, so an old Skill can name a file that later moved. On the 62 tasks where everything the old Skill names still exists, the pooled action gain is +0.22, against +0.23 on all 80. Information already available: old-version agents see the required token in their own command output in 42–50% of episodes, and where Qwen 3.5-27B's old agent had seen it, the new rule still raises action +0.19 [+0.08, +0.29]. Where the old Skill already contained the token (23 tasks), no action gain is detected.

On-demand loading. Re-run with only name and description in the prompt and the body at its path: the four-agent action gain falls from +0.23 to +0.12 [+0.06, +0.18], about 51% kept (CI 28–75%). The three open models keep ~38% (Qwen 3.5-27B +0.20 → +0.08, 9B +0.16 → +0.07, Devstral +0.36 → +0.13). Sonnet 4.5 keeps +0.19 under both modes, and on demand its old Skill works worse (automated final correctness 0.50 vs 0.67 preloaded) while the new one holds (0.69), so on-demand final correctness rises +0.19 [+0.03, +0.39]. The two agents traced open the Skill in 89% (Qwen 27B) and 88% (Sonnet) of new-version episodes, but the 2,000-character window displays the added rule in only 71% / 69% of reads; once displayed, Sonnet acts on it in 88% and Qwen in 61% (paired difference +0.24). With the body preloaded the figures are 81% / 62%, so Qwen's on-demand drop is not lower uptake once the rule is seen. This is the activation/adherence split of Harness Activation and Adherence measured on a single rule, and it adds a third gate that page does not name: the runner's display window between reading a file and seeing the line.

RQ3 — what it costs#

• Single answer. Input +18–19% on every open model, output shorter by 2–19 tokens ("a model told which command to run answers more concisely than one that hedges" — in juxt/allium a rule to run allium check cuts the answer 201 → 81 tokens and lifts compliance 0.07 → 0.99). Input+output +17% on open models, +24–27% on four Claude models; local latency falls 1.6–7.2% because decoding dominates. About 630–890 added tokens per additional compliant response.
• Agent episode. No detectable change from the revision (+0.8% mean; every CI spans zero) because the longer prompt is offset by fewer steps — agents with the new version run fewer commands, and with any body they list directories less. Carrying the body at all costs +37 to +57% per episode (50% mean), since mini-SWE-agent re-sends it every turn.
• Revision types. Over all 2,608 pairs, a rule-or-procedure revision adds a median +248 prompt tokens; a prose-only one +6.
• Overhead candidates. For each rule × model pair: compliance improves in 58%, falls in 5%, neither moves in 17%, and in 20% (CI 16.1–24.2%) the prompt grows with no detected gain. Of the 55 rules at least half the models flag as overhead, 27 are already followed (old compliance ≥0.5 without the rule) and 28 are ineffective (ignored either way). The authors' prescriptions differ: removal tests before deleting the first kind, a rewrite for the second. At five samples per condition, compliance moves in steps of 0.2, so "no detected gain" is not "no effect."

How much of this reaches real work#

Every RQ2 number is conditional on a request that needs the rule. The authors estimate how often that happens: of 1,454 later commits that touch a Skill's files or the paths it names, a stratified hand-labelled 100 re-weighted give 25% that need the rule (95% CI 14–40%). Under the stated scenario (full effect where needed, zero elsewhere), an in-scope request gains about 0.10 in single-answer compliance (0.25 × 0.41) and 0.06 in sandbox required action (0.25 × 0.23), while every request pays the revision's tokens. That dilution is the bridge to the transfer-task null below.

Reading it against the rest of the corpus#

• Against the maintenance null (Human-Governed Skill Maintenance). Shen & Hruschka find the latest version of 13 well-maintained Skills no better than the earliest (−0.09 on a 1–5 judged scale, CI [−0.28, +0.10]); this paper finds a revision raises compliance, action and blind-judged correctness. Not a contradiction — different estimands, both empirical. Here every probe is built so the revised rule applies, and the revision is a rule naming a command or path; there, transfer tasks are generated "without telegraphing [the skill's] conventions" and the edits are whatever the maintainers made, much of it process scaffolding. Wang et al.'s own 25% need-rate dilutes the conditional effect to ~0.06 action per in-scope request, which a judge panel at ICC 0.52 would not resolve. The two agree on the mechanism: S&H's sign split (output constraints helped, process scaffolding hurt) and this paper's split (new command/path tokens help, restatements and prose do not) both say benefit is a property of what the edit adds.
• Against the context-file null (Khatri, on Agent Context Files). Khatri's generic convention context moved correctness ~0 and his near-misses were implementation failures, not missing facts. This paper's rules are the opposite class — a specific command or path, not style — and the information was inferable: the required token existed in the repository and old-version agents saw it in their own output in 42–50% of episodes, yet stating it as a rule still added +0.19 action. That partially answers the "provably cannot infer" question on that page: what moves behaviour is not whether the agent could find the fact, but whether the instruction names it.
• Against vendor self-measurement (Skill Lift). NVIDIA's +41 Correctness compares with-skill to without on eval sets generated from the skill; here the comparison is version to version with a none arm, human-judged outcomes and repository-clustered CIs, and the body-vs-none compliance effect is +0.57. Same direction, independent instrument — though probes here are also generated from the rule under test, so the circularity is reduced (blind human judgement of artifacts), not removed. On cost, this is the first powered sign: carrying a body is +50% tokens per episode, where Skill Lift's two anecdotes ran −77% and +120%.
• Against Harness Shrinkage as Models Improve. For rules naming commands and paths, there is no shrinkage across 2023–2026 small open models or along the Sonnet 3.5 → 4.5 ladder (bound 0.062–0.077). That is consistent with the hub's own carve-out: scaffolding that supplies facts the model cannot know should not erode; scaffolding that supplies behaviour should. No reasoning model was tested.
• Against Instruction Compounding. Anthropic's Opus 5 guide says delete instructions that duplicate native behaviour because they overshoot. This paper's "already followed" rules (27 of 55 overhead rules) show no compliance change in either direction, and deleting a duplicate costs only −0.12 against −0.58 for a replaced rule — so pruning duplicates is cheap, pruning live rules is not. The instruments differ (a string check cannot see overshoot; the guide measures nothing), and the rule classes differ (named commands vs behavioural requests), so neither result supersedes the other.
• Against the pruning gap (Agentic Technical Debt, Agentic Work Systematization). Every lifecycle study finds Skills accrete and are rarely pruned. This paper prices both directions: an accreted rule that names something new pays; one in five rule-model pairs is pure overhead; and a deletion is obeyed almost as strongly as an addition, so "prune more" needs a per-rule test, which the authors propose as a pre-merge check that runs a revision's probes.
• Against Tool-Catalog Context Cost. Progressive disclosure is the skill-layer version of deferred tool loading. This is the first measurement of what deferral costs in effect, not tokens: about half of the action gain for the average agent, none for Sonnet 4.5, under a runner that truncates reads.

Connections#

• Human-Governed Skill Maintenance — the version-to-version comparison run on the task instead of the rule. Its −0.09 null and this page's +0.10 correctness gain measure different estimands, reconciled above by the 25% need rate and by both papers' "what the edit adds" split
• Harness Activation and Adherence — the two consumer-side gates measured for one added rule: Skill opened in 88–89% of episodes, rule displayed in 69–71% of reads, acted on in 61% (Qwen 27B) to 88% (Sonnet 4.5) once seen; on-demand loading keeps ~51% of the preloaded gain
• Skill Lift — the with/without ablation this page complements with old/new; gives the first powered sign for skill token cost (+37–57% per episode carrying the body, no detectable cost from a revision)
• Agentic Work Systematization — the adoption and lifecycle page; this is the first measurement of what a lifecycle edit does downstream, including a stale-reference control (+0.22 action on tasks with no stale reference vs +0.23 overall)
• Agent Context Files — the artefact family one level up and the co-change partner: Skill revisions touch CLAUDE.md and AGENTS.md more often than matched commits do; and a rule naming an inferable-but-unstated token still moves action +0.19
• Instruction Compounding — the redundant-instruction question from the measured side: "already followed" rules are inert under a string check, and deleting a duplicate costs −0.12 against −0.58 for a replaced rule
• Agentic Technical Debt — the complexity ratchet priced: 20% of rule-model pairs add tokens with no detected gain, and deletions are followed at ~¾ the strength of additions
• Tool-Catalog Context Cost — progressive disclosure as deferred loading, with the effect cost of deferral measured
• Harness Shrinkage as Models Improve — no generation interaction for rules naming commands and paths (bound 0.077; 0.062 on the Sonnet ladder)
• Anthropic — originator of the Agent Skill format and progressive-disclosure loading the paper's conditions replicate

Open Questions#

• Do prohibitions, conditional rules and permission guardrails follow the same old-vs-new effect? The literal filter kept 17% of candidate directives and under-represents exactly these classes (22% vs 32% prohibitions retained), because a string check cannot grade them. Falsifiable with a behavioural grader (did the forbidden command run?) on the paper's released probe and sandbox manifests.
• Does the on-demand retention gap survive a production harness that displays whole files? Here the sandbox shows only the first and last 2,000 characters of a read, so the added rule was displayed in only ~70% of reads, and the open models kept ~38% of the gain against Sonnet 4.5's 100%. Re-run the 80 sandbox tasks under Claude Code or Codex with full-file reads and report retention per model.
• Do reasoning models gain from a revision as non-reasoning models do? Every model here ran with thinking disabled or effort set to none; the generation-interaction null (bound 0.077) is therefore a statement about non-reasoning answers. Falsifiable by re-running the 160-probe subset with thinking on.

Sources#

• Agent Skill Evolution: How Revisions Affect Coding Agents — Jiajie Wang, Yutong Zhao, Tianlin Li, Huashan Chen, Jinfu Chen, Kebin Peng & Sen He (University of Arizona; CSU Long Beach; NTU Singapore; IIE Chinese Academy of Sciences; Wuhan University; East Carolina University), Agent Skill Evolution: How Revisions Affect Coding Agents, arXiv 2610.04832, 2026-10-04, 21 pp, empirical. Cited for §3 (dataset, probes, grader validation, settings, metrics), §4 + Table 2 + Figure 3 (RQ1), §5 + Tables 3–5 + Figures 4–5 (RQ2), §6 + Tables 6–7 (RQ3), §7.1 (threats) and §7.2 (the 25% need rate and maintainer implications). Replication package on figshare. Parse warnings, in this page's convention: ingest flagged table-collapse on 20 cells holding multiple values (100 / 200 / 250, 18% / 6%). All are false positives — the slash-separated values are the paper's own cell format (Table 1's compound row labels, Table 2's "Up / down" column, Table 4's "none / old / new" rows), and Tables 2 and 4 were verified cell-for-cell against pdftotext -layout. Table 4's two-line cells are merged in the raw (rate and Δ on one line) but the mapping is intact. The Figure 5 control table in the raw is partial (the "unrelated text" row is blank and a "0" floats below it); the figure-5 values quoted above are from the prose of §5.1.2. Figure 4 was viewed and matches the prose.