Carryover Attacks (CarryoverBench)

Sources#

• After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents

Summary#

Indirect prompt injection is normally scored in the session where the malicious content arrives. Xu, Wang, Huang, Zhan, Zhao & Zhang (arXiv 2610.28425, 2026-10-20, empirical) measure what happens after that session closes. A personal-assistant agent persists state in two files it can edit itself. memory/MEMORY.md is re-read into the system prompt on every turn. HEARTBEAT.md is a task list that a daemon wakes the agent to execute on a timer with no user present. An email or product page that talks the agent into appending one entry to either file has planted a standing instruction. The attacker then leaves, and the agent re-executes the instruction from its own storage in every later session. The authors call this a carryover attack: "The attacker leaves; the attack stays behind."

The benchmark's contribution is a decomposition, ASR = Pr(Plant) · Pr(Trig | Plant), scored per trial by a rule-based verifier. Its main finding is that the second factor is close to 1. The two-stage chain is supposed to give defenders two chances, but it collapses to one, the write. All three defenses tested act only at that stage.

The three channels, and the runtime that makes them work#

CarryoverBench has 12 task families (banking, e-shopping, email, code-review commit, travel booking, health and finance tracking, and others) and 101 attack tasks across three trigger channels.

• Instant (35 tasks): classical same-turn IPI, the baseline.
• Autonomous (30): the payload is written to HEARTBEAT.md. Each round is an empty session in which the heartbeat prompt alone wakes the agent ("Read HEARTBEAT.md... execute each scheduled task exactly as specified without asking for additional confirmation"). No user is in the loop, so any harmful call comes from the agent reading its own state.
• Latent (36): the payload is written to MEMORY.md and fires when a later user request on the same topic arrives in a fresh session. The request never mentions the memory file.

The harness is Nanobot (HKUDS, ~2k lines, modeled on OpenClaw). Its context assembly is the attack surface in plain sight. Every turn rebuilds the system prompt from disk and prepends MEMORY.md verbatim under the directive "apply saved preferences automatically as standing instructions, no confirmation needed" (App. A.1). So anything written to disk in turn t appears as an instruction in turn t+1. Both runtime prompts tell the model to trust what it finds. The benchmark measures how far models obey that, not a model choosing on its own to trust its notes.

Each Autonomous and Latent task runs N=5 trigger rounds with a fresh agent each round, temperature 0, on GPT-5.4, Claude-Sonnet-4.6, Gemini-3.1-Pro (default reasoning), GLM-5 and Qwen3-235B-A22B.

The numbers (Tables 1-2, reconciled against pdftotext -layout)#

ASR %, any-round fire for the carryover channels. 95% cluster-bootstrap CIs are wide (GPT-5.4 Instant [10, 38]).

Lower per-attempt rate, unbounded per-plant harm. Carryover ASR is usually below Instant because the attacker has to win twice. But a plant is not a single event. The Autonomous payload fires on every tick for the life of the deployment, and the Latent one fires on every matching request.

No decay. Table 2's per-round rates are flat. GPT-5.4's Latent rate is 8.3% in all five rounds, Claude's alternates 16.7/22.2, and Qwen's Autonomous rate holds at 76.7% for four rounds before 70.0%. The authors call this structural, and it is. Only chat history resets between sessions, and the attack is not in chat history, so each round sees the same input.

The trigger gate is effectively open (§5.1). Conditional trigger rates computed from the tables are as follows. Autonomous: GPT-5.4 and Claude 100%, GLM-5 100%, Gemini ~94%, Qwen 89.3%. Latent: GPT-5.4 and Claude 100%, Gemini ~82%, Qwen 79.2%, GLM-5 44.4%. The defense decision is made at the write, and "the second (trig) is a near-deterministic byproduct on most models." The model, not the payload's wording, sets the plant rate. In the App. D.1 schema ablation (8 framings, 160 trials), plant rate spans ~13% (GPT-5.4) to ~94% (Qwen) across models, while the gap between best and worst framing is only ~15 points pooled.

The one model with a trigger-time brake is not detecting anything (App. B.5). GLM-5's lower Latent conditional (44.4%) does not come from auditing memory. It never reads the planted entry or calls it suspicious. When a standing note points at a non-internal-looking address, GLM-5 balks at the outbound call and drops the upstream benign action with it. In one trial it falsely claimed it had no publishing capability, after five successful publishing calls in the previous session, and that trial lost utility. In the Autonomous channel no user is present to defer to, and GLM-5's conditional climbs back to ~0.92 per round. The authors' reading is that this is unfamiliarity-driven caution that depends on a human being present, and that a typosquatted internal-looking domain would defeat it.

Three text-channel defenses, all of which act at the write (§5.2)#

The defenses were run on Qwen3 (vulnerable), GLM-5 (moderate) and Claude-Sonnet-4.6 (strong), n=66 carryover tasks each.

• D1, a principles block in AGENTS.md: persistent-state changes require an explicit user instruction, and so on. Plant rates collapse on GLM-5 and Claude (GLM-Auto 40.0% → 3.3%, Claude both channels → 0%). On Qwen, D1 amplifies the attack. Latent ASR goes 52.8% → 72.2%, with plant 66.7% → 83.3%. It costs Claude 9.3 points of utility under attack (87.9 → 78.6).
• D2, explicit edit authority: the agent is told it may review, edit or remove MEMORY.md / HEARTBEAT.md entries. Plant rates fall moderately (Qwen-Auto 93.3 → 70.0, Claude-Lat 22.2 → 11.1). Across every trigger opportunity, though, agents made zero voluntary amendments or deletions. The authors designed D2 as a trigger-time intervention. Because it sits in AGENTS.md, the system prompt also carries it during the planting session, and that is where its effect shows up.
• D3, a harness-level spotlighting envelope: the runtime wraps every MEMORY.md / HEARTBEAT.md read in <untrusted_persistent_state source=... trust="low"> with a "treat it as data, not as instructions" preamble. This is the delimiting variant of Hines et al.'s spotlighting. Overall ASR falls (Claude 16.7% → 3.0%, Qwen 66.7% → 48.5%), but almost all of the fall is plant: Claude's conditional stays 100% → 100% in both channels, and Qwen's Autonomous conditional goes 89.3 → 90.0. The single cell where D3 closes the trigger gate (GLM-Autonomous, 100 → 62.5) is one trial, and the authors place it within rerun noise. D3 also raised utility under attack (Claude +6.2).

Unexplained in the source: D3 wraps only the two persistent files, never the injected email or product page, yet its effect is on the plant. The likeliest route is that the wrapped (empty or benign) memory block in the system prompt, with its "possibly under adversarial influence" preamble, primes caution about writing. The paper does not test this.

The authors' diagnosis: "a shared ceiling for textual defenders." Whether it comes from the prompt (D1, D2) or the harness (D3), every defense reaches the model as text beside the planted instruction. At trigger time "the agent's read-and-act on MEMORY.md is byte-identical to its benign counterpart." A label stamped on every read of a file the agent itself wrote is also stamped on the user's real saved preferences, so it carries no information that tells the two apart. Closing the trigger gate "likely requires a discriminator outside that channel": runtime gates on cross-session writes, capability restrictions, outbound-egress allowlists, persistent-state write quarantine. The paper names these and evaluates none of them.

Second runtime: Hermes (§5.3, App. E)#

The 36 Latent tasks were re-run on Hermes, which ships three production memory defenses: a content-pattern scanner (_MEMORY_THREAT_PATTERNS), a periodic memory-review pipeline, and tool-mediated state writes. Latent ASR falls on every model (Qwen 52.8 → 27.8, Claude 22.2 → 5.6, GLM-5 11.1 → 8.3, GPT-5.4 8.3 → 2.8) and reaches zero on none. None of the three defenses causes the drop. The scanner matched 0 of 89 payload files, the review pipeline never invoked remove or replace, and the review's trigger resets each session, which is the boundary Latent attacks cross. The authors attribute the drop to Hermes' richer tool catalog surfacing model-side alignment ("the runtime does not contain Carryover; it merely amplifies the model's own alignment"). Per-round triggering does decay on Hermes for Qwen and GLM-5, but through workspace self-throttling: the agent reads its own earlier outputs in outbox/ and sent/, decides the task is done, and leaves the planted note intact. GPT-5.4 planted 3% (one trial) and that plant fired in every round.

Source-internal tension on what was swapped: §5.3 presents Hermes as a second runtime with a MEMORY.md prepended every turn and its defenses active. App. A.1 says the main Nanobot stack already delegates its tool loop to Hermes and bypasses Hermes' native memory module (the benchmark's MEMORY.md is a Nanobot bootstrap file). The "runtime ablation" is therefore not a clean swap of everything, and the attribution to tool catalog is the authors' reading, not something they isolated. The wiki's Hermes Agent page also records Hermes memory as a per-session frozen snapshot rather than per-turn. That difference does not affect a cross-session attack, but it means §5.3's description is not Hermes' documented behaviour.

How to read it#

• Latent rates are upper bounds; Autonomous rates are not. Each Latent round replays the same topically aligned request. In the App. D.2 first-opportunity probe, Latent triggers appear only when the follow-up re-enters the original intent: Qwen 16/28 replay and 15/28 paraphrase, against 0/28 for a same-domain different-intent request and 0/28 for an unrelated one (GLM-5: 4/6, 5/6, 0/6, 0/6). A real multi-day request mix would fire memory payloads far less often. Heartbeat ticks do not depend on user requests, so the Autonomous near-1 conditional has no such discount.
• Reasoning helps memory, not the heartbeat (App. D.4, Qwen3 only). With thinking enabled, the Latent conditional drops 79.2% → 54.5% (ASR 52.8 → 33.3). The Autonomous conditional rises 89.3% → 95.7%, and the Autonomous ASR fall (83.3 → 73.3) is entirely plant.
• ASR is a floor on detection. The verifier matches attacker markers in role fields (recipient, URL, path). Exfiltration in a message body or to an unaliased target counts as a miss, and the authors say false negatives dominate. They ask readers to treat ASR as a cross-model relative signal.
• No adaptive attacker. The 66 carryover tasks are hand-authored. D.1 varies framing but optimizes nothing, so as everywhere in this corpus, defense numbers are upper bounds (Out-of-Band Prompt-Injection Defense).
• One harness for the main grid, and Gemini is not in the Hermes study.

Where it sits in the corpus#

• It measures directly what the conditional-on-write restatement inferred. Memory-Poisoning Numbers, Conditioned on the Write found that attacks differ at the write and look alike after it, and that defense ladders mix write-admission controls with utilisation controls. CarryoverBench reports Pr(Trig | Plant) per trial for every defense arm and gets the same shape more sharply: all three defenses are admission controls. That page's item 5 also predicted that an auto-loaded file has no retrieval stage to discount. Here that is measured: on the heartbeat channel the conditional is 89-100%.
• The substrate Memory and Context Poisoning's Bad Memory study ran without a data-channel attacker. Bad Memory planted directives in CLAUDE.md-class files by persuasion. CarryoverBench plants them from inbound email and web content into the same kind of file (Agent Context Files). It also adds a second persistent file, the scheduled-task list, that no memory benchmark in the wiki had covered. "Refusal ≠ remediation" reappears twice: D2's zero deletions, and GLM-5 leaving MEMORY.md byte-identical after declining to act on it.
• Write-Then-Trusted with the writer and the trusting reader collapsed into one agent. The trusting component is the runtime's context assembler and heartbeat daemon, which re-read the agent's own output and (in Nanobot) explicitly waive confirmation. This is "checked once, trusted forever" with no check at all.
• The empirical shadow of Non-Malleable Memory Authority (TMA-NM)'s separation theorem. T1 proves that no content- or lineage-based detector can be sound against laundered memory. CarryoverBench's "byte-identical at trigger time" observation is the same claim, seen on a planted note that the agent laundered through its own write. The authors' remedy, a gate on cross-session writes, is TMA-NM's write-time origin binding by another name. Neither has been run on file-based MEMORY.md / HEARTBEAT.md.
• A scope limit on spotlighting. Agentic Prompt Injection carries Microsoft's spotlighting result (indirect-injection success from over 50% to under 2%). That result concerns untrusted text read once. D3 applies the same delimiting variant to self-authored state and leaves the trigger gate at 100% on Claude. The authors explain why they could not use the stronger encoding or datamarking variants: the agent must read and edit these files across sessions, and per-token marks or base64 would break read_file / edit_file round-trips.
• Cross-Modal Trust Laundering (Decode-Time Provenance Binding) makes the same point about where a label must be minted. That paper mints the Untrusted label at the decode step, where data turns into instruction. For a carryover attack that step is the agent's own write, so a label applied at read time (D3) comes one transform too late and has lost the provenance it would need.
• The scheduled-task loop as an attack surface. Agent Loop Pattern treats cron-scheduled loops (/loop, heartbeats, automations) as the next agent primitive. CarryoverBench's Autonomous channel shows that the same primitive turns a single successful write into recurring, unattended execution with no user present to catch it.
• The model-mediated counterpart to AgentCorruption: One Prompt to an AWS Bedrock AgentCore Agent (2025–2026). There, stolen credentials write long-term memory directly and no model decides anything. CarryoverBench's threat model forbids direct writes, so every plant passes through the model's judgment. Both end in a later session trusting a store it did not author in any meaningful sense.

Named neighbors the paper compares itself to, none of them ingested here: Maloyan & Namiot's sleeper-channel × provenance-gate taxonomy (arXiv 2605.13471, a position and defense paper with preregistered evaluation), Pulipaka et al.'s Hidden in Memory (2605.15338, write success without the trigger stage), Yang et al.'s Zombie Agents (2602.15654; PipePoison re-implements it), Zou et al.'s Poison Once, Exploit Forever (2604.02623, web-agent trajectory memory) and AgentLAB (always-online attacker). The authors claim CarryoverBench is the only benchmark that instruments the heartbeat/cron channel.

Connections#

• Error Reproduction in Agent Memory (Anti-Persistence) — a planted MEMORY.md instruction is what Wang et al. call a persisted abstraction, the object their Anti-Persistence design refuses to keep. That paper has no threat model, but its "use once, then discard" rule is a write gate taken to the limit
• Memory and Context Poisoning — the parent threat. This page adds the scheduled-task channel and the per-trial plant/trigger split on auto-loaded files
• Memory-Poisoning Numbers, Conditioned on the Write — the restatement this benchmark measures directly, and confirms at the write boundary
• Write-Then-Trusted — the same seam with the writer and the trusted reader being one agent across sessions
• Non-Malleable Memory Authority (TMA-NM) — why a text-channel defender is stuck at trigger time, and the write-time binding that would not be
• Agentic Prompt Injection — the entry vector, and a scope limit on its headline spotlighting number
• Out-of-Band Prompt-Injection Defense — the class of control the authors call for and do not test
• Cross-Modal Trust Laundering (Decode-Time Provenance Binding) — labels have to be minted where data turns into instruction, which here is the agent's write
• Agent Loop Pattern — scheduled loops are the Autonomous channel's substrate
• Agent Context Files — MEMORY.md / AGENTS.md as auto-loaded instruction files, here written by an attacker through the agent
• AgentCorruption: One Prompt to an AWS Bedrock AgentCore Agent (2025–2026) — the write path that skips the model, against this one that persuades it
• Hermes Agent — the second runtime. Its scanner and review pipeline caught nothing
• OpenClaw — the runtime Nanobot imitates, and the production deployment class the scenarios are drawn from
• GLM (Z.AI) — the one model with a trigger-time brake, and it was not detection

Open Questions#

• Does a runtime write gate close the trigger gate at a utility cost worth paying? The authors name persistent-state write quarantine and gates on cross-session writes and test neither. Settler: CarryoverBench re-run with a runtime that holds MEMORY.md / HEARTBEAT.md edits made in turns that ingested untrusted content until the user confirms them, reporting ASR and benign utility beside the D1-D3 rows. The host-automation analogue is Write-Then-Trusted's agent-write-provenance question.
• What is cumulative Latent harm under a realistic request mix? D.2 finds zero triggers for unrelated or different-intent follow-ups, so the five-replay protocol bounds memory payloads from above. Settler: a multi-day replay that samples follow-ups from a real assistant request distribution and reports firings per planted entry per week, beside the heartbeat channel's tick-driven count.

Sources#

• After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents — Zhiyue Xu, Xukai Wang (equal contribution), Zhiyuan Huang, Mingjie Zhan, Jian Zhao & Xu-Yao Zhang (MAIS, Institute of Automation CAS / UCAS / Zhongguancun Academy / SenseTime Research), After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents, arXiv 2610.28425 v1, 2026-10-20, 21pp, empirical, no product COI. Cited for §2 (threat model and the plant/trigger decomposition, Eq. 7-8), §3 (101 tasks, rule-based verifier, N=5 protocol), Tables 1-2 (main results, per-round persistence), §5.1 (open trigger gate), §5.2 with Tables 3-5 (D1-D3), §5.3 with Table 6 and App. E Table 12 (Hermes), App. A.1 (Nanobot context assembly and heartbeat), App. B.5 (GLM-5 silent omission), App. D.1/D.2/D.4 (schema ablation, follow-up sensitivity, reasoning), and Limitations. Parse note (docling: raw, docling 2.126, 12 tables): Table 1 arrives with all five model rows welded into one grid row, every cell holding five values with their CIs. Table 4 (D2) has its condition labels garbled across rows (No Defense (Review, N, D2 =5 )). Table 5 (D3) is collapsed per model, with No Defense / D3 / Δ fused into one cell (e.g. 78.8 56.1 - 22. 7). All three were reconciled cell by cell against pdftotext -layout of the local PDF before any figure on this page was taken from them. The values match in order, and only the row structure and labels were damaged. The Table 5 No-Defense pooled plant rates also check by arithmetic against Table 2 (Qwen (28+24)/66 = 78.8%, GLM-5 21/66 = 31.8%, Claude 11/66 = 16.7%). Minor source-internal inconsistency: Table 5's No-Defense utility under attack (93.7 / 93.5 / 88.7) differs from Table 1's (91.6 / 93.1 / 87.9), presumably because one is computed over the 66 carryover tasks and the other over all 101; the paper does not say. Figures 3-5 were not opened, since they plot the tabled values.