Agent Systems, in order.

Open questions 60 open

• Agent Context Files
  • How should context files and bounded memory files interact when they disagree? Memory is lossy and cache-delayed; the context file is authoritative but static. Which wins, and when?
• Agent Harness Engineering
  • At what codebase scale does the AGENTS.md-as-table-of-contents approach need to be replaced with more sophisticated context routing?
  • How generalizable are these web-app-focused findings to other domains (scientific research, financial modeling)?
• Agent Quality Flywheel
  • Both demo cycles fixed agents with *instruction-level* bugs and showed large one-cycle gains. What does the loop look like on failures that need tool, memory, or architecture changes — does "several iterations before metrics move" dominate in practice?
  • The custom rubric is authored by the same coding agent that will later propose fixes. Metric *choice* is upstream of grading — does decoupling need to extend to who defines the metric, not just who scores it?
  • Synthetic User Simulator scenarios bootstrapped the whole first cycle. How much of the 21%→5% delta survives on real-traffic distributions (the representativeness gap [[production-sourced-evaluation]] names)?
• Build for the Next Model
  • How do you tell a "wait for the model" gap from a durable-harness gap *before* the next release? Get it wrong and you either ship vaporware or build a crutch you'll delete.
  • The bet depends on a reliable release cadence and a forecastable capability curve ([[task-time-horizon-scaling]]). What happens to "build for the next model" if model improvement stalls (the [[recursive-self-improvement|stalled-but-diffused]] future)?
  • Does the strategy generalize outside frontier labs, who have privileged visibility into the next model? An external team is betting on a release it can't see.
• Claude Code Auto Mode
  • What false-positive rate does the classifier have on routine-but-aggressive refactors (e.g., large-file renames, `rm` of build artifacts)?
  • How well does the classifier generalize to custom tools / MCP servers where it lacks environment context?
  • Is the classifier's decision boundary documented/stable enough for security-sensitive orgs to certify, or is it effectively a black box whose behavior drifts with updates?
  • Does extending auto mode to API users change its calibration — is the classifier retrained for automation-heavy use, or held constant?
  • Compared to OS-level sandboxing (mentioned in [[claude-code-best-practices]] alongside auto mode), what's the defense-in-depth story? When should both be layered?
• Claude Code Best Practices
  • What's the optimal CLAUDE.md length before instructions start getting lost? Is there a measurable threshold?
  • How does the Writer/Reviewer pattern compare to agent-to-agent review (as in OpenAI's Codex workflow)?
  • When does subagent overhead exceed the benefit of context isolation?
• Client-Side Agent Optimization
  • How does combination-level optimization interact with continual model releases? If Claude Opus 4.7 ships next month, does the full Pareto frontier need re-running, or do warm-started bandits adapt cheaply?
  • At what pipeline depth does the combinatorial search become intractable even for Arm Elimination? The paper tests up to ~81 combinations; production pipelines with 5+ roles and 10+ candidate models each blow past that.
  • Does the "weak planner + strong solver" pattern generalize, or is it specific to HotpotQA's delegation dynamic? Recommender-critic, drafter-editor, and retriever-generator topologies might invert.
  • What's the right way to re-evaluate when the tool environment changes? AgentOpt assumes fixed tools — adding or removing a tool potentially invalidates the whole frontier.
  • Is there a cheap per-call classifier that can predict which combination will win on a given query, avoiding combo-level evaluation entirely?
• Codex App Server Protocol
  • How does the App Server protocol compare in detail to MCP? Both expose tools to a model, but App Server is *inside* the Codex runtime while MCP is *outside*. When does each win?
  • Is there a public schema registry so external orchestrators can target specific App Server versions without `generate-json-schema`?
  • The "dynamic tool calls (experimental)" caveat — what's the stability roadmap? Symphony depends on this for its security model.
  • How well does the protocol handle multi-modal turns (image inputs, screenshot attachments)? The spec is text-focused.
  • Is there an analogous protocol on the Claude side, or is Claude's equivalent exclusively the Agent SDK + tool-use API? Comparing the two would clarify when "drive an existing CLI" beats "build on the SDK."
• Context Window Smart Zone
  • Does the smart-zone marker scale with model size, or is it bounded by attention architecture? Pocock observes "the dumb zone has become less dumb lately" but pegs it at 100K through 2026.
  • How should harnesses surface remaining smart-zone budget to the user — token count, percentage, or a richer signal?
• Deep Modules for Agents
  • How big is "deep enough"? Pocock's example modules are several hundred LOC; Ousterhout's textbook examples are larger. There's a sweet spot; not articulated.
  • For ports/adapters codebases, does the deep-module advice transfer cleanly? The "small interface" is the port; the "large behavior" is the adapter. Probably yes, but not exercised in source.
  • Refactor cost vs benefit: when is "improve-code-base-architecture" worth running on a working repo?
• Deep Research Agents
  • DRACO grades single-turn interactions only. How much of real deep-research value is in the multi-turn loop (clarifying questions, follow-ups) that the benchmark doesn't yet measure?
  • Factual accuracy is the weak axis everywhere — is the fix better retrieval, better verification-in-the-loop, or a tool-grounded check the way Lean grounds proof search?
• Failures That Look Like Success
  • Is "internal state correct, final message stale" a general LLM-agent failure signature (state/utterance divergence) or an artifact of session-state architectures like ADK's? A cross-framework tally would tell.
  • What fraction of production agent failures are silent-contract violations vs. loud errors? The 14/15 and 3/4 numbers are demo-sized; telemetry-scale data ([[production-sourced-evaluation]]) could ground the class.
• Harness Shrinkage as Models Improve
  • Does *all* prompt scaffolding eventually migrate into the model, or does some remain — e.g. organization-specific style, security rules, brand voice?
• Latent vs. Deterministic Space
  • Tan asserts the "wrong side" diagnosis covers most AI-engineering bugs. Does any incident/failure taxonomy (agent postmortems, eval failure analyses) actually classify failures by computation-locus, and what fraction lands in each side?
• LLM-as-Compiler Knowledge Base
  • At what scale does the no-vector-database approach break down? Karpathy's ~100 articles fit in context, but what about 1,000+?
  • How to handle conflicting information across sources during compilation?
  • What's the optimal granularity for concept articles — one concept per article, or clustered by theme?
  • How effective is the synthetic training data → fine-tuning pipeline in practice?
• Loop Engineering
  • Osmani's cost caveat is unquantified: at what token budget does a continuously-running loop stop paying for itself, and how do you instrument that? (Cf. [[agent-loop-pattern]]'s "who owns the budget when the model schedules its own loops.")
  • If `/goal`'s stop-check is itself a model, what verifies the verifier? The maker/checker split pushes the trust problem up a level, not away.
• MCP and Computer Use
  • MCP security model: as the playbook prescribes wiring MCP into Salesforce, Gmail, Calendar for solo founders, the attack surface scales with adoption. **Now addressed** by [[zero-trust-for-ai-agents]] (tool poisoning, rug pulls, the first in-the-wild malicious MCP server) — see "MCP as a security surface" above. Open residual: how does a solo founder realistically *run/host and self-sign every MCP server* the framework recommends, given that the appeal of MCP was zero-integration-effort? **Sharpened by [[mcp-tool-poisoning|ShareLock]]:** the cheaper alternative to self-hosting — *scan the tool descriptions with a guard model* — is information-theoretically defeated by threshold fragmentation, so the lightweight mitigation doesn't hold and the burden falls back on run-your-own-server or downstream action-layer authorization. **And [[mcp-tool-poisoning|Agentjacking]] shows run-your-own-server *itself* isn't sufficient:** when the server is a legitimate observability platform relaying attacker-injected data (fake Sentry errors), self-hosting/vetting the server catches nothing — the untrusted input rides in on its data, so the residual burden falls squarely on the downstream data/action layer (provenance tracking + an out-of-band action gate), not on server hygiene.
  • How does Cowork's computer-use guardrail compare to Claude Code's auto-mode classifier? Different deployment context, possibly different risk profile.
• Optimizer–Evaluator Decoupling
  • Does decoupling need to extend upstream to metric *design*? An optimizer that authors its own rubric has a subtler channel to game than one that merely reads scores.
  • How much independence is enough — different model family, different vendor, different modality of check (model judge vs. compiled test vs. production telemetry)?
• Parallel Agent Orchestration
  • Summed-overlap runtime can exceed 24h/day — it measures *agent* effort, not *human* attention. What is the human's actual oversight load per concurrent agent, and where does it saturate ([[ai-brain-fry]])? **Sharpened:** [[configurable-human-participation|HAS-Bench]] reframes the *shape* rather than measuring the load — in a controlled (LLM-simulated) benchmark the value of human input is configuration-dependent and *non-monotonic* (a right-timing / right-channel / right-authority sweet spot; more agency brings diminishing and sometimes negative returns), so per-agent oversight is likely a returns-curve with a peak, not a linear cost that hits a wall — but it is single-human, single-task, so it does not measure real concurrent-oversight load.
  • Concurrency is measured over one week. Is 5+-agent management a stable practice or a burst around specific large tasks?
• Repository Exploration Subagent
  • **Does the gain survive better main models?** The same-model-exploration result suggests the architectural benefit is somewhat model-independent, but the trained-explorer margin may erode as frontier models get cheaper and better at staying in their smart zone unaided. The bitter-lesson question is unresolved.
  • **Prune vs. don't-pollute.** SWE-Pruner removes context after the fact; FastContext avoids accumulating it. Are these complementary (prune the solver *and* delegate exploration) or substitutes? Not tested together.
  • **How small can the explorer go?** The authors flag 1.7B / 0.6B as future work — if the recipe holds, the explorer becomes nearly free and the architecture dominates.
  • **Generality beyond Mini-SWE-Agent.** Only one (deliberately minimal) main-agent scaffold is tested; richer harnesses with their own memory/subagent orchestration may already capture part of the benefit or interact differently.
  • **Patch-derived reward leakage.** Training the explorer's reward on the gold patch's file/line ranges risks overfitting to where fixes *landed* rather than where evidence *lives*; the F1-vs-recall behavior partly mitigates this, but the proxy is imperfect.
• Ticket-Driven Agent Orchestration
  • What's the right granularity for ticket size when the unit is "what one agent does in one workspace"? The post implies "much larger units of work" become viable, but how does that interact with the `agent.max_turns` limit (default 20)?
  • How do you prevent a ticket-extension cascade when agents file follow-up tickets liberally? Is the only governance check human triage at the `Todo`-state queue?
  • Does this pattern generalize to non-software work (research, ops, content)? The DAG dependency model and prompt-as-policy file should transfer; the per-issue workspace doesn't obviously.
  • When an agent gets a ticket "completely wrong" (mentioned in the post), how is the lesson fed back into the system? Symphony's answer is "add guardrails and skills" — what's the institutional process for that?
  • How does ticket-driven orchestration interact with sprint planning / OKRs / roadmap work that operates on aggregates of tickets? Does the abstraction collapse when tickets are scoped that small?