Context Window Smart Zone

Sources#

• Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems
• Full Walkthrough: Workflow for AI Coding — Matt Pocock
• Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models
• Self-GC: Self-Governing Context for Long-Horizon LLM Agents
• Uncle Bob on Software Fundamentals in the Age of AI

Summary#

LLMs do not degrade linearly as context grows; they degrade quadratically because attention relationships scale O(n²) with token count. Matt Pocock (citing Dex Horthy of Human Layer) frames this as a smart zone / dumb zone split: the first ~100K tokens of any session is the smart zone where the model performs well; beyond that the model gets "dumber and dumber" regardless of advertised window size. Practical implication: context budget is a real, hard resource — and the agent harness is responsible for keeping individual sessions within the smart zone.

The constraint#

"Every time you add a token to an LLM, it's kind of like you're adding a team to a football league. The number of matches goes up quadratically."

"It doesn't matter whether you're using 1 million context window or 200K, it's always going to be about [100K]. It starts to just get dumber."

— Matt Pocock

The 1M-token context windows shipping in 2026 don't move the smart zone — they "just shipped a lot more dumb zone." Long context is useful for retrieval (find a fact in five copies of War and Peace) but not for reasoning (write code that depends on all of it).

Memento metaphor#

Each session is a fresh start. There is no memory across sessions; the model resets to the system prompt every time. This is a constraint but also a feature — clearing context restores smart-zone behavior cheaply. Persistent state must live somewhere the next session can read it (repo, filesystem, a the index-style catalog).

Compaction is worse than clearing#

Claude Code's /compact command summarizes the running session into a smaller history. Pocock prefers /clear:

• Compacted history accumulates "sediment" — distortions and lossy summaries — that degrades subsequent work
• Clear-and-restart returns to a known-clean baseline (the system prompt)
• The cost of clearing is paid back by working in the smart zone

The disagreement isn't universal — many developers like compaction because it preserves continuity. The right call depends on whether your task can be resumed cleanly from a written record (then prefer clear) or needs in-flight conversational context (then compaction wins). (the binary framing superseded 2026-08-03 by Self-GC: Self-Governing Context for Long-Horizon LLM Agents; the underlying observation still holds — see below)

Measured (2026-08-03): what summarization loses, and a third option#

Pocock's argument is practitioner-opinion; Self-GC (Xiaohongshu, arXiv 2607.00692, empirical) measures the same thing on 332 production-derived agent sessions and the practitioner intuition survives — with a sharper mechanism and one correction:

• "Sediment" has a name. What summarization loses is not blurriness in general but exact evidence, locators and live handles, behavioral contracts (user corrections), verbatim source text, and current live state. A summary "preserves narrative state but hides exact evidence, locators, and editable artifacts."
• The choice is not binary. A third policy — govern the run at object granularity, folding bulky payloads to byte-exact recoverable sidecars rather than summarizing them — covers exactly the case where both clear and compact are bad: the task needs in-flight context and exact artifacts.
• Cutting harder is not free. Heuristics that remove 62–70% of Hard Set prefix tokens land at 54.55–69.70% "no-impact" (higher is better); Self-GC removes 43.95% and reaches 84.85%. A harness that stays inside the smart zone by pruning aggressively trades a quadratic-attention problem for a missing-dependency problem.

The cliff has a number (added 2026-08-04). The most vivid measured instance of Pocock's sediment claim is one the corpus reaches at second hand: compressing an 18,282-token context to 122 tokens in a single unvalidated step dropped task accuracy from 66.7% to 57.1% — below the no-context baseline. Summarizing was worse than sending nothing, because the summarizer had no way to know what would be needed downstream. Provenance matters here: the figure is Zhang et al. 2025 (Agentic Context Engineering, arXiv 2510.04618) as cited by Maximem's ACM paper (arXiv 2607.21503, empirical, sole author with a total vendor COI — but the COI attaches to that paper's own product claims, not to a number it is quoting from third-party work the wiki has not read directly). The generalization the citing paper draws is the one to keep, and it is the same distinction Context Lifecycle Management draws with a mechanism: crude summarization buys linear cost by surrendering fidelity, and the failure is not a degraded answer but a confident wrong one.

Measured (2026-08-04): retrieval survives far past the marker, and what breaks near the ceiling is refusal#

The ~100K marker above is a claim about reasoning, and Pocock's own carve-out is that long context stays useful for retrieval. Eliav 2026 (arXiv 2607.19257, empirical) is the corpus's first controlled measurement of that carve-out: a deterministic, contamination-free 512,000-token synthetic corpus (8,780 uniquely-named fictional entities, generated from a fixed seed with no LLM involved, so chance-level recall is ~0), sliced into a 2k→512k ladder, rendered in four content-identical formats, probed for recall, false-premise sycophancy, and fabrication of facts never stated. Five models, 5,520 calls each at full context, 30,480 scored responses. The carve-out survives, and three findings sharpen it.

1. Recall is flat to 64–128k, then breaks format-dependently. At 2k, 16k and 64k every model in every format sits at 0.98–1.00 recall accuracy and no format discriminates. Separation first appears at 128k. So the smart-zone marker is not a retrieval marker — a model can still pull a specific unguessable fact out of 64k of undifferentiated text essentially perfectly, and Gemini Flash is still at 0.93–1.00 at 512k.

2. The wall is per-model, not per-token. Claude Haiku at 128k is the largest format spread in the dataset — plain text collapses to 0.383 recall while markdown, prose and table sit at 0.817–0.867, a 48.4pp gap driven by one format alone. Meanwhile Sonnet 5's format spread nearly doubles from 256k (11.7pp) to 512k (20.0pp) while Gemini Flash's stays flat (5.0pp → 6.7pp) across the identical token range, with both models carrying the same documented 1,000,000-token ceiling. Spread magnitude tracks proximity to a model's own effective ceiling, not absolute token count. This is the strongest evidence in the corpus that a single universal marker — 100K or otherwise — is the wrong shape for the constraint; the number is a per-model property that has to be measured, and the advertised window does not predict it.

3. The failure mode near the ceiling is refusal, not hallucination. This is the finding the page did not have, and it inverts the usual monitoring posture. Fabrication of unstated facts is exactly zero — 0 of 5,760 absent-fact probes, every model, every rung, every format. Direct sycophantic agreement with a stated false premise peaks at 8.3% in a single cell and sits at or under 3% in the large majority. What climbs, steeply, is outright refusal to answer the false-premise probe:

Model	2k	16k	64k	128k	256k	512k
Claude Haiku	0.000	0.142	0.704	0.896	—	—
Sonnet 5	0.154	0.058	0.317	0.217	0.579	0.788
Gemini Flash	0.004	0.000	0.004	0.042	0.350	0.250
Qwen 27B	0.008	0.000	0.125	0.362	—	—
Qwen 35B	0.038	0.096	0.088	0.154	—	—

Refusal also contaminates the recall numbers in a way worth knowing: at Sonnet 5's 512k rung, three in four of its incorrect markdown responses are a literal "insufficient information" non-answer rather than a wrong guess, so plain and table's apparent recall advantage there is partly a lower propensity to refuse rather than better comprehension. A harness that monitors only for wrong or agreeable answers near the top of a window is watching the wrong failure.

Format is not free, and the cost can flip the choice. Rendering the same facts costs 1.221× plain for prose, 1.258× for markdown, 1.367× for a table — stable across every rung. At Claude Haiku @ 128k, prose and table tie on raw accuracy (0.867 each) and overhead breaks the tie in prose's favour. The paper's own bound on this: outside the two cells with a genuine accuracy gap, every format is within noise and picking the cheapest is a tiebreak, not an efficiency gain.

Caveats. Sole author, single lab, preprint. For the three short-context models the "ceiling" the refusal curve climbs toward is the largest rung tested (128k), not a measured wall, so "approaching its own ceiling" is doing looser work there than for Sonnet 5 and Gemini Flash. Cross-rung comparisons in this design are only valid against the paper's fixed 20-question anchor set — it caught one apparent Gemini Flash 256k→512k improvement (0.900 → 1.000) as a question-composition artifact — and the refusal climb and the ranking flips both survive that check. And the fabrication null cannot fully separate "verified the fact's absence" from "declined to answer anything at this rung," since the absent probe scores declining-to-answer as correct; the paper says so itself.

Implications for harness design#

1. System prompt budget. Anything always-in-context comes off the smart-zone budget. "I have seen people put 250K tokens [in the system prompt], then you're just going into the dumb zone before you can even do anything." Keep CLAUDE.md / AGENTS.md as a table of contents, not an encyclopedia (see Agent Harness Engineering on AGENTS.md as ToC).
2. Sub-agents preserve parent context. A sub-agent runs in its own context window; only its summary returns. Pocock's grill-me skill ran a 93.7K-token sub-agent yet his main session still had ~25K tokens unused.
3. Fragment work into many sessions. Loops (see Agent Loop Pattern) and vertical slices (see Vertical Slice Tracer Bullets) work because each iteration starts fresh in the smart zone.
4. Reviewer should run in fresh context. If the implementer used 80K tokens in the smart zone, asking it to review its own work pushes the reviewer into the dumb zone. Cleared context = smart-zone reviewer (see Deep Modules for Agents on push-vs-pull and reviewer placement).
5. Push vs pull instructions. Always-in-context instructions cost smart-zone tokens; pull-on-demand (skills) costs nothing until invoked. The token budget is not the only ceiling on the always-in-context block: instruction count has its own, independent of length — see Instruction Compounding.
6. Budget for refusal near the ceiling, not hallucination. The measured failure at the top of a window is the model declining to answer (up to 89.6% in one model), not inventing something. Alerting and evals aimed only at wrong-or-agreeable answers will read clean through it, and a rising non-answer rate will look like a quality regression rather than a context-budget symptom.

Status-line token counter as an essential tool#

Pocock recommends a status-line widget showing the exact running token count of each session — without it, developers don't know when they're approaching the dumb zone. He treats this as "absolutely essential information."

Independent corroboration, and the harness consequence (Martin, 2026-08)#

Robert C. Martin arrives at this page's constraint from a different direction and gives it a different name (Uncle Bob on Software Fundamentals in the Age of AI, 2026-08-19, practitioner-opinion). He went looking for why the coding rules in his prompt stopped being followed, and landed on lost in the middle:

"As the context window builds up inside the model the stuff at the very beginning and the stuff at the very end have more prominence than the stuff in the middle… anything you say at the very beginning is going to get shoved into the middle if it's long. So maybe the first three sentences you put at the beginning will remain as priority, but the 50th and the 80th sentence in there, they're gone."

The two framings are not the same claim and it is worth keeping them apart. The smart-zone framing is about total occupancy — performance degrades as the window fills, wherever your content sits. Lost-in-the-middle is about position — the middle of the window is served worse than either end, at any occupancy. Martin's practical prescription follows from the second: it is not "keep the session short" but "keep the instruction block short," because a long instruction block manufactures its own middle. "The key with agents is to trim that initial prompt down to its absolute minimum so that you can get as much of it as possible into its priority." Instruction Compounding measures a related but distinct ceiling — all-rules-obeyed compliance reaching zero around 80 simultaneous instructions, independent of format — which would predict Martin's observation without any positional term at all. Nobody has separated the two.

The design consequence he draws is role-scoped, short-lived agents. This is the same conclusion this page reaches from clear-over-compaction, arrived at from the instruction side:

"When you focus the agents down to a single task, you're keeping the context window under control. The lost in the middle problem becomes much less of a problem. So you can pile a few more, not a lot more, but a few more rules up at the top and they'll tend to follow them better. You can also set up a system where the agents are born, do the task, and die so that the next one comes in with a clean context."

He prices the pattern honestly: 10–15 seconds of startup per agent, "and then it's got to figure out its whole context all over again." See Parallel Agent Orchestration for the five-role pipeline this produces, and Latent vs. Deterministic Space for the corollary he draws — that anything you cannot afford to have decay should not be in the window at all.

Marker drift worth noting. In this conversation the smart-zone boundary is given as "the first 150k tokens," against the ~100K this page has carried since 2026-05. The caption track does not label speakers, but the phrasing ("I refer to this as… it's not my term, that's Dex Horthy's term, I stole it") and the attention-dilution explanation place it with Matt Pocock, the same practitioner who supplied the 100K figure. Neither number is measured; both are practitioner feel, and the movement is in the direction model context handling has been going.

Connections#

• Robert C. Martin (Uncle Bob) — independent arrival at the same constraint via lost-in-the-middle, with the minimize-the-prompt prescription and born-do-die role agents

• Reviving Impractical Quality Tools — his response to the decay: move the durable rules out of the window and into checkers

• Context Lifecycle Management — the measured alternative to both clearing and compaction: govern the active context as indexed objects with byte-exact recovery, and price the commit against prefix-cache disruption

• Instruction Compounding — the other ceiling on always-in-context material, measured in the same paper and denominated in instruction count rather than tokens: a prompt block collapses past ~80 simultaneous rules whatever its length or rendering

• Scale-Dependent Prompt Sensitivity — the format half of the long-context result: which rendering wins reverses between models and between adjacent rungs of the same model, so there is no portable "use markdown for injected context" rule

• Matt Pocock — popularizer of the smart-zone framing

• Agent Harness Engineering — system-prompt minimalism and AGENTS.md-as-ToC are restatements of the smart-zone principle

• Agent Loop Pattern — fragmenting work to stay in smart zone is why loops are powerful

• Vertical Slice Tracer Bullets — keeping each task small enough to fit in smart zone

• Design Concept Grilling — the grilling session uses a sub-agent so the parent context stays small

• Deep Modules for Agents — clearing-before-review is a smart-zone discipline

• Harness Shrinkage as Models Improve — the smart zone may grow ("the dumb zone has become less dumb lately") but quadratic attention still constrains it

• AI Brain Fry — human-side analog of the smart zone: oversight has its own degradation curve past capacity, mirroring attention degradation past ~100K tokens

• Interaction Models — continuous audio/video at 200ms granularity accumulates context fast; TML names long-session context management as an open problem — the same constraint in a new modality

• HTML as the New Markdown — the human-attention analog: a reader degrades past some volume of undifferentiated markdown the way a model degrades past ~100K tokens; HTML raises the human's effective smart zone by spending tokens on legibility

• Agentic Technical Debt — founders' persistent-context discipline (CLAUDE.md) competes with smart-zone budget; over-long context files become their own problem

• Authority and Audit Survive Abundance — this page's effective-ceiling/refusal evidence as the unpriced third leg of "free" long context: dollar price and capability price diverge — though it is the one leg of the retrieval defense that model improvement erodes rather than structure protecting

Open Questions#

• Does the smart-zone marker scale with model size, or is it bounded by attention architecture? Pocock observes "the dumb zone has become less dumb lately" but pegs it at 100K through 2026. Partially answered 2026-08-04 by Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models (empirical) — neither cleanly, because the premise of a single marker doesn't hold. Degradation onset is a per-model effective ceiling that the advertised window does not predict: two models sharing the same documented 1,000,000-token ceiling diverge sharply in format-spread growth over the identical 256k→512k range. It also splits the question by task — retrieval holds at 0.98–1.00 through 64k and past 128k for some models, well beyond the 100K marker, which is consistent with Pocock's own retrieval-vs-reasoning carve-out. The paper measures no reasoning task, so whether the ~100K reasoning marker is architectural remains untested.
• When sparse-attention or memory-augmented architectures ship, does the smart zone become a soft constraint? Partially answered (2026-08-12) — the trigger event landed, and the first evidence points the other way. Sparse attention is now, in Ren et al.'s words, "widely deployed in long-context serving stacks" — so this is no longer a prediction. Their dense-calibrated audit (empirical) supplies the mechanism that argues against the soft-constraint hope: block selection does not merely make more context affordable, it severs cross-block attention, and an ablation isolating one probe block from cross-block communication collapses its behavioural influence from 4.48 logits to exactly zero across all 1,536 units, with sparse partial connectivity sitting between the two. Since the smart zone is a claim about reasoning over the window rather than retrieval within it, and reasoning is precisely what needs blocks to talk to each other, a cheap window bought by discarding blocks is not the same object as a large window. Three reasons this is only partial: the outcome is a logit-margin influence proxy with no reasoning or task-accuracy measurement anywhere; all models are 7B–8B; and the audit never varies context length, so it says nothing about where a sparse model's own effective ceiling sits. Retagged out of the Predictions ledger now that the architectures have shipped.
• How should harnesses surface remaining smart-zone budget to the user — token count, percentage, or a richer signal?

Sources#

• Full Walkthrough: Workflow for AI Coding — Matt Pocock — primary articulation
• Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems — Gaurav Dadhich (Maximem), arXiv 2607.21503, 2026-07-23, empirical, sole author with a total vendor COI (the paper's benchmark subject is its author's own product). Cited here for one number only, from Table 1 and §3.2: the 18,282 → 122 token compression dropping accuracy 66.7% → 57.1%, itself a secondary citation to Zhang et al. 2025 (arXiv 2510.04618), which the wiki has not read directly. None of that paper's own product or benchmark claims is used on this page; they live on Context Lifecycle Management with the COI stated in full. Its Table 3 is cell-collapsed and Table 4 scrambled in the raw parse; neither is cited anywhere
• Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models — Netanel Eliav, arXiv 2607.19257, 2026-07-21 (empirical, sole author, single lab, not peer reviewed): §3 the Book of Veyra corpus and context ladder, §5.2 recall accuracy and Table 7's non-ceiling cells, §5.4 the fabrication and sycophancy nulls, §5.5 and Table 8 the refusal climb, §5.6 the anchor-set methodology caveat, §7 cost-adjusted accuracy. Tables 3, 7, 8 and 9 were reconciled and are clean; the paper's Table 1 model roster is cell-collapsed in the raw markdown and is not cited (see the Sources note on Instruction Compounding), so per-model context ceilings here come from the paper's prose
• Uncle Bob on Software Fundamentals in the Age of AI — Robert C. Martin with Matt Pocock, 2026-08-19 (practitioner-opinion; auto-caption transcript): lost-in-the-middle as the reason prompt rules decay, minimize-the-prompt as the prescription, born-do-die role agents as the harness consequence; also the source of the 150K smart-zone figure, unlabeled speaker