Stale Guidance: Do Agents Route Around It, and What Does It Cost?

Sources#

• Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix

The question#

From Agentic Work Systematization's Open Questions: "Does a stale skill measurably degrade agent task outcomes, or do models increasingly route around outdated instructions (Harness Shrinkage as Models Improve)?" The bullet already carried two partial answers: the Shen & Hruschka transfer null (2026-09-22) and the Wang et al. conditional effect (2026-10-21). This synthesis adds Xu & Wu (arXiv 2610.30571, 2026-10-22, empirical), compiled today as Stale Instruction File Precedence. It checks that paper against the raw itself, not only the wiki page.

The question contains two claims that different evidence settles, so they are answered separately.

Half 1: do models route around outdated instructions? No, and the trend is not toward it#

Current frontier agents follow stale guidance even when they can see that it is wrong. Xu & Wu built 40 Python projects in which the instruction file asks for one convention and every existing module uses the other. Claude Code (2.1.302, Sonnet 5.5) and Codex CLI (0.163.0, GPT-6 Sol) added a feature to each, and 90.1% of convention choices (649 of 720) matched the stale file, against 4.2% matching the code (raw §5.1). Claude Code followed the file in 86.4% of its choices and Codex in 93.9%. This is not a perception failure. Claude Code named the file–code conflict in its final reply in 79 of 80 runs and still followed the file in 86.2% of its choices in those runs (raw §5.2). It reported the code as the part out of line. "Routing around" would mean the model weighs visible current evidence over an outdated instruction. The paper offers the agent exactly that evidence, the agent notices it, and it declines to use it (Stale Instruction File Precedence, "precedence, not perception").

The skill-layer evidence agrees. Wang et al. hold the request fixed and swap an old SKILL.md for its revision (Skill Revision Effects, RQ2). Agents follow what the version in front of them says. A newly added rule raises sandbox required action by +0.23, and deleting a rule lowers compliance with it by 0.27–0.33, about three-quarters of what adding one gains. A model that routed around its skill would show neither effect.

"Increasingly" is the claim the evidence most directly rejects. Wang et al. test twelve older-versus-newer within-family comparisons and find no significant generation interaction. The data-derived bound is 0.077, and 0.062 along the Sonnet 3.5 → 3.7 → 4.5 ladder. The one significant step has the older model gaining less (Skill Revision Effects). Newer models depend on the current rule as much as older ones did, so they are not learning to infer it without the skill. Xu & Wu's agents are a generation past Wang et al.'s and follow stale files at 86–94%. That is consistent with Harness Shrinkage as Models Improve's own carve-out, as Skill Revision Effects reads it: scaffolding that supplies facts or conventions the model cannot know does not erode, and only behaviour-request scaffolding migrates into the model. A stale convention is a stale fact delivered as a command. Shen & Hruschka's weaker non-reasoning solver also showed no version effect, so their null was not a strong model compensating for an old skill (Human-Governed Skill Maintenance).

The exceptions, so they are not lost. Claude Code followed the stale file in fewer than half its choices for two of 16 kinds of convention, money (~32%) and configuration (~14%) (raw §5.1, Figure 5b). Money is the one kind whose convention changes behaviour (integer cents versus floating point). That fits Claude Code giving correctness-bearing conventions more weight against the file, though the paper neither explains nor tests it (Stale Instruction File Precedence). This is the only routing-around the corpus has measured. It covers two convention kinds in one agent, so it does not change the overall answer.

Why it is settled and not merely suggested. The routing-around reading predicts code-following when the evidence is in view. Xu & Wu put the evidence in view by construction (the agent edits the conflicting modules), confirm from the replies that it was seen, and measure the opposite outcome on both vendors' current agents. Wang et al. supply the cross-generation trend. The mechanism the authors propose, an instruction ranking above tool output under instruction-hierarchy training, is not isolated: there is no channel-swap arm (Stale Instruction File Precedence, "How strong the mechanism claim is"). That leaves why agents do not route around stale guidance open. It does not leave open whether they do.

Half 2: does stale guidance measurably degrade task outcomes? Conditionally yes. Unconditionally, unmeasured and probably small#

When a request touches the stale rule, outcomes degrade measurably, in three forms:

1. Correctness of the result (skills). On requests that need the revised rule, an old Skill yields lower blind-judged final correctness, −0.10 pooled over three agents, and less required action, −0.23 over four (Skill Revision Effects; the per-agent CIs for Qwen 3.5-9B and 27B exclude zero, while Sonnet 4.5's is underpowered rather than null). Wang et al. checked whether "stale" in the narrow sense (an old Skill naming a file that later moved) explains this. It does not: +0.22 on the 62 tasks with no missing reference against +0.23 on all 80. The cost comes from the old version lacking the current rule, which is the substantive meaning of a stale skill.
2. Breaking working code (instruction files). Without the precedence sentence, Codex CLI rewrote pre-existing modules to match AGENTS.md in 20 of 80 runs, a median of 198 lines removed from 9 files, and the project's original tests then failed in 17 (raw §5.4). Codex's pass rate was 78.8% without the sentence against 100% with it. Every failing run was a rewrite. Over both agents the gap is 89.4% against 100% (raw §5.5). Claude Code rewrote nothing in either condition, so this test-measured cost belongs to one agent.
3. Off-convention new code (instruction files). Almost all of the 90.1% stale-file choices are new code that does not match the project. No test catches it, because no test checks logging against print (Stale Instruction File Precedence, via Layered Supervision). The paper counts it as review cost and, for money, a correctness risk, citing Mohammadi et al.'s floating-point arithmetic under a stale rule. That is degraded quality that a pass/fail outcome metric does not register.

Pass/fail metrics hide part of the cost. Form 3 is invisible to task success. Co-Installed Skill Conflicts measures the same blind spot from another direction: when a rival skill displaces the installed one, use of the installed skill falls 19.9 pp and fidelity on its exclusive requirements falls 5.6 pp, while task completion is flat (+1.9). Harness Activation and Adherence draws the general lesson. A following rate measures obedience to the artifact and counts as benefit only while the artifact is current. Whether stale guidance "degrades outcomes" therefore depends on what the outcome measures. On tests it shows only where an agent rewrites code. On conventions and skill-specific requirements it shows on nearly every request that touches the rule.

Unconditionally, the cost dilutes to a size current studies cannot resolve. Wang et al. estimate that only about 25% (CI 14–40%) of later in-scope commits need the revised rule. That turns the conditional −0.23 action into about 0.06 per in-scope request (Skill Revision Effects, "How much of this reaches real work"). Shen & Hruschka's transfer test, latest against earliest version of 13 maintained skills, came out at −0.09 on a 1–5 scale (95% CI [−0.28, +0.10]), with a judge-panel ICC of 0.52 (Human-Governed Skill Maintenance). A 0.06 effect sits inside that interval. The null and the conditional effect are different estimands, not contradictory results, and both papers name a natural task stream as the missing test.

Persistence multiplies the conditional cost. A stale instruction file is reloaded every session, so "the same mistake recurs in every task, for every agent and every team member, until someone corrects the file" (raw §2). Error Reproduction in Agent Memory (Anti-Persistence) measures the self-generated version: a faulty stored abstraction regenerates through the trajectories it shaped, and discarding synthesized guidance beats persisting it in 4 of 4 settings. Neither result measures the cumulative cost of one human-authored stale line over a real project's lifetime.

What carries across from instruction files to skills, and what does not#

The new evidence is about CLAUDE.md/AGENTS.md, not SKILL.md. The question asks about skills. What transfers:

• The direction transfers. Wang et al. independently show agents obeying whichever Skill version they are given, and the paper's own field cases include a stale Supabase SKILL.md (React 18 guidance for 13 days after the React 19 migration, raw §2, Figure 2).
• The magnitude may not. Xu & Wu's explanation rests on the file being auto-injected as an instruction, with activation at 1.0. A Skill is often loaded on demand through a file read, which is the tool-output channel the paper says models rank lower. Wang et al. find on-demand loading keeps only about 51% of a revision's action gain on average, though all of it for Sonnet 4.5 (Skill Revision Effects). If the channel explanation is right, an on-demand stale Skill should be followed less often than a stale CLAUDE.md. No study has run a stale Skill against conflicting code in both loading modes.

Status for the open question#

• Settled, and retirable as a split: do current models route around outdated instructions, increasingly or at all? No. They follow stale guidance against visible contradicting code at 86–94% (Xu & Wu, two vendors, current generation, 14 of 16 convention kinds). The generation trend is flat for skill rules (Wang et al., bound ≤0.077). The money/configuration exception for Claude Code is the only counterexample measured.
• Settled conditionally: on requests that touch the stale rule, outcomes degrade: −0.10 correctness (skills), 90% off-convention code, and test breakage through Codex rewrites (17 of 20).
• Still open: the unconditional outcome cost of a stale skill on a natural task stream, and whether on-demand Skill delivery softens the stale-file effect. Settlers: Wang et al.'s v_old/v_new pairs or Shen & Hruschka's pairs on an independently sourced, natural task set, or Xu & Wu's public 40-project design rerun with the stale convention in an on-demand SKILL.md instead of an injected instruction file.

Verdict on the oq-match hit#

Jev scored Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix as a possible full answer (p_full 0.51). Against the raw, it is partial. It settles the routing-around half outright, together with Wang et al. for the generation trend. It adds a test-measured outcome cost, but only for one agent's rewrites, only for instruction files, and only on synthetic projects with consistent conventions and single unattended feature tasks (raw §8). It says nothing about skills delivered on demand or about the unconditional cost.