Stale Instruction File Precedence

Sources#

Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix

Summary#

An instruction file describes the project as it was when someone wrote it. When the code moves on and the file does not, an agent receives two conflicting accounts of the same convention: the file says print(), every module uses logging. Xu & Wu (Georgia Tech / CU Boulder, arXiv 2610.30571, 2026-10-22, empirical) measure which account wins. The file wins:

90.1% of choices (649 of 720) matched the stale file and 4.2% the existing code. Claude Code (2.1.302, Sonnet 5.5) matched the file in 86.4% of its choices and Codex CLI (0.163.0, GPT-6 Sol) in 93.9%. The file prevailed for both agents in 14 of the 16 kinds of convention.
Noticing the conflict does not change the outcome. Claude Code named the conflict in its final reply in 79 of 80 runs, and 86.2% of its choices in those runs still matched the file. It read the code, saw the difference and reported the code as out of line: "The existing concat.py, reader.py, writer.py and columns.py still use logging, pathlib, str.format() and absolute imports, which go against CLAUDE.md."
One agent breaks working code to satisfy the file. Codex CLI named the conflict in 2 of 80 replies. In 60 runs it followed the file silently. In the other 20 it rewrote existing modules to match AGENTS.md (a median of 198 lines removed from 9 pre-existing files, across 12 of the 40 projects), and the project's original tests then failed in 17. In the Figure 7 case it replaced a Column enum with string constants, so the tests could no longer import Column. Claude Code rewrote existing code in neither condition.
One sentence reverses it. With this sentence in the user-level file (~/.claude/CLAUDE.md, ~/.codex/AGENTS.md), 91.4% of choices (658 of 720) followed the code:

"When a repository's documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code."

That is +87.2 pp (95% CI 83.6–90.9, from resampling projects): Claude Code 7.5% → 94.4%, Codex CLI 0.8% → 88.3%. Both agents reported the conflicts in all 160 runs. No Codex run rewrote code, and the test pass rate rose from 89.4% to 100%. Codex accounts for all of that gap (78.8% → 100%), and every one of its failing runs was a rewrite. The sentence works without anyone first finding the stale line.

The vault had treated stale guidance as a perception failure: "a human senses the mismatch; an agent treats it as current" (Context Smells), and an out-of-date SKILL.md "is executed rather than read" (Agentic Work Systematization). For Claude Code this paper measures something narrower. The agent does sense the mismatch and ranks the file above what it sensed. The failure is precedence, not perception. Codex's silent compliance is still consistent with the older reading, but its replies are the only evidence there, and a reply that does not mention the conflict does not show the agent did not see it.

Why the file wins: three reasonable properties, composed#

The authors attribute the behaviour to three properties of current agents. Each is defensible alone (§3):

Instruction files describe code in the imperative. "Use print() for progress messages" is phrased as a command but records how the code looked. When the code changes, the description becomes false and the imperative mood stays.
Agent software delivers the file as an instruction. Claude Code inserts CLAUDE.md into a user message directly after the system prompt, under the preamble "Codebase and user instructions are shown below. Be sure to adhere to these instructions. IMPORTANT: These instructions OVERRIDE any default behavior and you MUST follow them exactly as written." Codex CLI places AGENTS.md in a user message headed "AGENTS.md instructions for <path>" inside an INSTRUCTIONS tag, without the emphatic line. Both put the user-level file first, then the project file, in the same message. The code arrives only as the output of a file-reading tool (Figure 3).
Models rank instructions above tool output. That is the instruction hierarchy (Wallace et al. 2024; the 2026 IH-Challenge dataset), trained in so that text inside a web page or file cannot redirect the agent. It is the main structural defence against Agentic Prompt Injection, and the property that page records models as lacking ("LLMs cannot reliably distinguish information from instructions"). The same ranking hands a stale line in the file priority over the code it once described.

The fix follows from the same ordering. The precedence sentence arrives as an instruction too, so following the code becomes what an instruction asks for, and the conflict becomes one between two instructions with an explicit tiebreak. It addresses only documentation–code disagreement, so the file's other lines (build commands, test commands) keep their effect.

How strong the mechanism claim is. Weaker than the headline suggests. The experiment tests two predictions the explanation makes: agents follow the stale file even after noticing the conflict, and the sentence reverses the outcome. Both held. But the design never moves a source between channels. There is no arm that delivers the stale convention as a tool read, or the code's convention as an instruction. A competing account fits the same data: agents follow any explicit precedence rule they are given, and absent one they default to the file for reasons that need not involve the tool-output rank. The paper also argues that Claude Code's emphatic preamble is not needed, because Codex has no preamble and followed the file more (93.9%). That comparison changes agent, model and delivery format at once, so it is suggestive only. Treat the channel explanation as the authors' leading hypothesis, consistent with the results and not isolated by them.

Design#

40 synthetic Python projects, each with 3–8 working modules, a passing test suite, an instruction file and a feature request that needs 4–5 convention choices. The code uses one convention consistently and the file asks for the other. Feature requests describe behaviour only and never mention conventions.
16 kinds of choice (Figure 4, viewed): messages (logging/print), paths (pathlib/os.path), exception type, error signalling (raise vs (value, error)), test file names, test style (pytest/unittest), records (dataclass/namedtuple), function names (snake/camel), optional types, imports (absolute/relative), generic types, string formatting, docstrings, money (integer cents/Decimal), configuration (dict/dataclass), fixed value sets (Enum/string constants). Each kind appears in 11–12 projects, in both directions, so a model's prior preference for one convention favours neither outcome.
Two agents × two conditions × 40 projects × 2 runs = 320 runs, non-interactive with auto-approved edits, fresh copy per run, and a dedicated config folder holding the condition's user-level file. Claude Code ran with CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 and --strict-mcp-config.
Measures. A script classifies each added choice as file, code or neither. A run counts as a rewrite when it removes ≥20 lines from pre-existing Python files. Original tests are restored after the run and run alongside held-out feature tests. A blinded Claude Opus 5.5 judge reads shuffled final replies under a criterion fixed before the first run and records whether each names the conflict. It runs in --safe-mode, which disables instruction files, skills, hooks and MCP. Results are reported per agent and per kind, citing the authors' own item-level-regression argument that an aggregate can hide opposite movements.

The two exceptions: money and configuration#

Figure 5b (viewed) puts every kind for both agents above 75% file-following without the sentence, except two for Claude Code: money ≈32% and configuration ≈14%. Codex follows the file on both (≈82% and ≈77%). The paper reports the exception and does not explain it. Money is the one kind on the list where the file's convention can change behaviour: Mohammadi et al., cited in §2, saw agents write floating-point money arithmetic under a stale rule while the code used integer cents. That fits Claude Code weighing a correctness-bearing convention differently from a stylistic one. The paper does not test it, and configuration (dict vs dataclass) does not fit the reading as neatly. Record it as an unexplained per-kind split.

Where stale files come from#

§2 gathers field evidence. None of it is new measurement, and all of it is cited secondary evidence the vault has not ingested:

73.8% of AI configuration files in 441 company repositories were committed once and never modified (Denisov-Blanch et al., arXiv 2608.25241).
A documentation-consistency checker flagged references to code that no longer exists in the instruction files of 23.0% of 356 public repositories (Treude & Baltes, "context rot", arXiv 2606.09090).
Three dated cases from 2026 (Figure 2, viewed). A Supabase SKILL.md told agents for 13 days after the React 19 migration that Studio "currently uses React 18 — skip these patterns". A Supabase CLAUDE.md said "pnpm 10" for 2 days after the move to pnpm 11. A Codex AGENTS.md asked agents for 24 days to update a narrative doc that had already been deleted. Maintainers corrected all three eventually.

These sit beside the vault's own lifecycle numbers: on Agentic Work Systematization, 53% of reused skills are never modified after the copy. On Human-Governed Skill Maintenance, where files are maintained, correction is 38% of substantive edits and deprecation 1 of 254. Claude Code's own documentation asks users to review the file periodically and prune outdated lines. The authors' objection is the practical one: that depends on someone noticing which line went wrong. The sentence takes effect without that.

What it changes on neighbouring pages#

Adherence is not unconditionally good. Harness Activation and Adherence treats following a loaded artifact as the benefit gate, and its numbers top out around 0.75–0.89 for strong models. This paper puts adherence at ~90% to an artifact that is wrong. That is the same gate, scored against the code rather than the file. A high harness-following rate measures obedience to the file, not correctness, and is only as good as the file is current.
Khatri's correctness null and this result are compatible, and together they say what a context file is. Agent Context Files carries Khatri's ablation: removing a consistent AGENTS.md costs no correctness, and the file "changes how the agent works, not whether it succeeds". Here a file that disagrees with the code decides nine conventions in ten. When file and code agree, its influence is invisible because the code already says the same thing. When they disagree, it dominates. Correctness moved only through Codex's rewrites (−10.6 pp pass rate without the sentence).
The derived policy/fact split is the fix, now with a measurement. When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time ruled that on policy the context file wins and on facts neither cache wins: verify against the repo, then repair the stale cache. Agents as shipped apply the policy rule to both, because a description of the code phrased as a command is indistinguishable from a rule. The precedence sentence, and the paper's model-maker recommendation ("'Run the tests before you commit' is a rule… 'We use print for progress messages' describes the code and can be checked against it", Figure 8), are that split implemented. That synthesis was written from first principles in 2026-07. This is the first controlled evidence that the fact half has to be stated explicitly to take effect.
The executable layer catches what the preventive layer gets wrong. Layered Supervision's practitioners keep a lint rule beside every steering-file convention "so that the executable check still catches violations when the steering artifact is outdated". Here the original tests caught 17 of Codex's 20 rewrites, so the executable layer worked. It flagged none of the ~650 stale convention choices in new code as such, because no test checks logging against print. That is the gap the lint duplicate exists to close.
The same scope ordering, used as a lever. On Co-Installed Skill Conflicts, install scope silently decides which same-job skill runs (Claude Code prefers the personal skill over the project's) and the user is not told. Here the user-level file's position, read first and on equal footing with the project file, is what lets one personal sentence override a project-wide stale rule. Both papers find scope precedence doing more work than any documentation describes. Per the authors, Claude Code's memory documentation explains file-vs-file conflicts and says nothing about file-vs-code.
A sibling persistence failure in memory. Error Reproduction in Agent Memory (Anti-Persistence) shows a stored rule regenerating from the trajectories it shaped. A stale instruction file has the same persistence profile with no feedback loop needed: it is reloaded every session, so "the same mistake recurs in every task, for every agent and every team member, until someone corrects the file."
It is the converse of impose-values-not-disciplines. Impose Values, Not Disciplines finds agents ignoring a prompt's process instruction (write tests first) and reverting to their own order. Here they obey a content instruction against the visible evidence. Process instructions bend to the model's habits and convention instructions do not, at least not against tool output.

Practice#

Users: add the sentence to the user-level file once. It applies to every project. It is the cheapest intervention in this corpus with a measured effect this large, and its cost is in the first open question below.
Teams: the sentence also makes both agents report every conflict (160 of 160 runs). That turns each agent session into a free staleness detector for the file, which is the maintenance signal Agent Context Files' playbook section says review should supply and nothing instruments.
Agent vendors: put the rule in the system prompt, so it reaches users who never edit their own file.
Model trainers: train models to check descriptive lines against the code rather than ranking all of the file as instruction.

Limits#

Synthetic and uniform. The code uses its convention consistently, so "follow the existing code" always has one answer. Real repositories mid-migration mix both conventions, and there the sentence gives no tiebreak.
Only the case where the file is wrong. Every project makes the file the stale source. The opposite case, a team that updated CLAUDE.md to start a migration while the code is still legacy, is where the sentence would make the agent resist the intended change. It is not tested, and it is the obvious cost of the fix.
Two runs per cell, one model per agent, both current-generation (Sonnet 5.5, GPT-6 Sol). Python only, single unattended feature tasks.
Mechanism not isolated (see above): no channel-swap arm, and the cross-agent preamble argument is confounded.
The conflict judge is same-family with one agent under test (Opus 5.5 judging Claude Code replies), with no agreement statistics reported. Blinding to condition and agent is real. Conflict reporting is the only judged quantity, though. Convention choices, rewrites and test outcomes are script-measured.
COI: none declared, and no author affiliation ships either agent. Data and code are public (github.com/WenJing95/stale-docs-data). The authors cite seven of their own 2026 papers, which is context for the related-work framing and no finding rests on them.

Connections#

Agent Context Files: the artifact class this measures, and where Khatri's correctness null and the maintenance playbook live; the stale-file case is the one those sections assume away
Harness Activation and Adherence: the adherence gate scored against the code instead of the file. ~90% adherence to a stale artifact shows that a following rate measures obedience, not benefit
Context Smells: stale guidance, measured. The agent does sense the mismatch (Claude Code 79/80) and defers to the file anyway, so the smell is a precedence failure rather than the perception failure Houck describes
Code as Source of Truth: Fung's rule as something the agent does not apply by default. The repo is the source of truth only if an instruction says so
Agentic Work Systematization: "stale guidance is executed rather than read", refined: read, flagged and executed anyway
Human-Governed Skill Maintenance: the maintenance loop that would remove stale lines, measured as rarely correcting and almost never deprecating
Layered Supervision: the lint-beside-the-steering-file duplication, and why tests caught the rewrites but not the stale new code
Co-Installed Skill Conflicts: scope precedence deciding outcomes no documentation describes, there between two skills and here between the user-level and project files
Agentic Prompt Injection: the instruction-over-tool-output ranking that defends against injection is the same ranking that lets a stale file beat the code
Error Reproduction in Agent Memory (Anti-Persistence): a sibling persistence failure. A wrong stored rule reloads every session until a human or a reset removes it
Impose Values, Not Disciplines: the converse case. Agents override process instructions with their own habits but obey convention instructions against the evidence

Open Questions#

Does the precedence sentence cost anything when the instruction file is right and the code is legacy, as in a migration the team started by editing CLAUDE.md first, or in a codebase that mixes both conventions? Settler: rerun the 40-project design (data package public) with a file-correct arm and a mixed-convention arm, scoring adoption of the intended convention.
Is the file's win caused by the instruction-vs-tool-output channel, or by the absence of any stated precedence? Settler: a channel-swap arm in which the stale convention is reached only through a file read (no auto-injection) and the code's convention arrives as an instruction. If code-following rises without the sentence, the channel explanation holds.

Derived#

When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time: the policy/fact split this paper's fix implements; written before any measurement, which this page supplies
Stale Guidance: Do Agents Route Around It, and What Does It Cost? — carries this page's 90.1% stale-file obedience over to the skill question: it settles that agents do not route around stale guidance, and flags on-demand Skill delivery (the tool-output channel) as where the magnitude may differ

Sources#

Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix: Xiaonan Xu (Georgia Tech) & Wenjing Wu (CU Boulder), Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix, arXiv 2610.30571, 2026-10-22, 14 pp, empirical. Cited for the abstract, §3 (delivery formats and the hierarchy argument), §4 design and Appendix A, §5.1–5.5 results, §6/Figure 8 recommendations and §8 scope. PDF-derived raw (docling 2.126.0, grade excellent, 0 tables, so the table checks do not apply). Every number here comes from prose or a figure caption. Figures 1, 2, 4, 5, 6 and 8 viewed. Figure 5b's per-kind values (money ≈32%, configuration ≈14% for Claude Code) are read off the dot plot, not stated in the text, and Figure 6 confirms the run split (Claude Code without sentence: 71 followed-and-reported, 6 no-majority; Codex with sentence: 79 followed code, 1 no-majority). The §2 field statistics (73.8%, 23.0%) are cited from other papers, not measured here. COI: none.