Agent Systems, in order.

Open questions 157 open

• Agent Context Files5 open
  • WaitWill the role split converge on Hermes's explicit project/personality separation, or stay folded into a single file as in Claude Code? A separate SOUL.md-style personality layer seems strictly better for multi-project users but adds a file to maintain.
  • WaitIs there a natural ceiling on the layering (project → workflow → spec → constitution), or does each new autonomy surface spawn another context-file tier?
  • SourceDoes the universal system-prompt slot cost anything? Every vendor on this page injects context files into the system prompt, and the only controlled measurement of that choice (prompt design at scale) finds placement is a larger lever than format with a model-specific sign — helping two models, hurting two. Falsifiable cheaply: render the same CLAUDE.md / AGENTS.md into the first user turn instead and measure adherence per model. (Genkit's skills middleware is a fourth vendor making the same choice — frontmatter metadata injected into the system prompt at init — which widens the premise without touching the question.)
  • SourceDoes context the agent provably cannot infer move correctness, where generic convention context does not? Khatri's null is scoped to naturalistic style-guide content on repositories the agent can read in full, and his failure triage says the gating deficit is implementation skill. The discriminating experiment is his own stated gap: rerun the ablation with purpose-built, task-specific context encoding a fact absent from the codebase (an undocumented external API contract, a deployment invariant, a "this test is flaky for reason X" note) and see whether near-misses flip. If they don't, the practitioner implication hardens from "generic files don't pay" to "context files don't pay for correctness at all." Partially answered: NVIDIA SkillEvaluator runs that arm on proprietary product knowledge and reports +41 Correctness / +39 Effectiveness across 300+ skills — the predicted direction, from a design whose eval set is generated from the skill under test, with no confidence intervals and an unnamed grader, so it moves the prior without closing the question. The version that would close it is an independently-sourced task set.
  • SourceMetadata-injection discovery relocates the instruction-count ceiling onto the skill catalog rather than removing it: every installed skill's description is resident from initialization, so a large enough skills/ directory should floor adherence before any skill body loads. How many resident descriptions does that take, and does use_skill selection degrade before or after all-rules compliance does? Falsifiable with prompt design at scale's harness pointed at N skill frontmatters instead of N rules.
  • ResolvedHow should context files and bounded memory files interact when they disagree? Memory is lossy and cache-delayed; the context file is authoritative but static. Which wins, and when? Answered: When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time — split by disagreement type. Policy: the context file always wins — it is the human-reviewed, git-versioned high-integrity channel, and agent-written memory's recency cannot confer authority (a memory item contradicting policy is indistinguishable from staleness or poisoning, per the TMA-NM laundering theorem). Facts: neither wins — both are caches over reality; verify against the repo/live state (the code-as-source-of-truth arbiter) and repair the stale cache. Always: log the conflict for the lint/pruning pass (the deviations-log pattern) instead of silently breaking the tie, and let writes flow only down the integrity ordering — memory never modifies the context file; the context file legitimately bounds memory.
• Agent Harness Engineering3 open
  • WaitHow does architectural coherence evolve over years in a fully agent-generated system?
  • SourceAt what codebase scale does the AGENTS.md-as-table-of-contents approach need to be replaced with more sophisticated context routing?
  • SourceHow generalizable are these web-app-focused findings to other domains (scientific research, financial modeling)?
  • ResolvedDoes a single general-purpose coding agent outperform a multi-agent architecture with specialized testing, QA, and cleanup agents? Answered: Single General Agent vs. Multi-Agent Coding Architecture — no single winner as posed; a single general agent overtakes a bespoke hand-engineered multi-agent system as models improve (The Bitter Lesson), but a monolithic-context agent loses to role separation (fresh-context explorer/reviewer + independent grader), which persists because it fixes structural constraints (quadratic attention, Goodhart), not model weakness.
• Agent Loop Pattern3 open
  • WaitWhen the model schedules its own loops (4.7 behavior), who owns the budget? Boris answered "the model just decides" — but that pushes cost discipline into the model's training, not the harness.
  • WaitDoes a loop with a smart enough model still need a Kanban backlog, or does the model choose its own next task from raw goals?
  • NoteLoop output review is now Matt Pocock's confessed bottleneck — "we just need to be ready to be doing more code review."
• Agent Quality Flywheel4 open
  • SourceBoth demo cycles fixed agents with instruction-level bugs and showed large one-cycle gains. What does the loop look like on failures that need tool, memory, or architecture changes — does "several iterations before metrics move" dominate in practice?
  • SourceThe custom rubric is authored by the same coding agent that will later propose fixes. Metric choice is upstream of grading — does decoupling need to extend to who defines the metric, not just who scores it?
  • SourceSynthetic User Simulator scenarios bootstrapped the whole first cycle. How much of the 21%→5% delta survives on real-traffic distributions (the representativeness gap Production-Sourced Evaluation names)?
  • SourceDoes a weight-space flywheel actually beat the frontier model it distils from, or only draw level with it? The only curve in the corpus reaches the frontier reference baseline by 0.4 judge points at the largest dataset size tested, with no error bars, no seeds and no intervals, and the claimed overtake rests on an un-charted GRPO stage after it. The falsifiable version: the same distillation sweep with seeds and confidence intervals, reporting the SFT-only and SFT+RL arms separately against the frontier reference.
• Agent-Authored Harness Optimization5 open
  • SourceDoes agent-authored harness evolution actually beat simple test-time scaling, and does it generalize to held-out tasks? Partially answered — and the two halves now point opposite ways. Test-time scaling: answered, negatively. rethinking harness evolution evaluation (empirical) supplies the full-suite budget-matched arm this bullet named as missing — four methods at K = 5 on Terminal-Bench 2.1 across three frontier models, with and without unit-test feedback — and harness evolution finishes below plain parallel sampling everywhere and below do-nothing on average without unit tests (67.4 vs 68.2), with its pass@5 gain traceable to multi-attempt selection rather than pass@1 capability. Held-out generalization: contested, not settled. harnessbank harness self evolution credits six of seven sealed per-domain tests at z ≥ 1.96 (+9.2 to +15.4pp); Wang et al.'s 45/10/34 split on Terminal-Bench 2.1 returns +0.6pp average and +0.0 on GPT-5.4. ~~The proposed discriminator is baseline headroom~~ (superseded 2026-08-13: score headroom fails on the decisive pair — Wang et al.'s 68.2 baseline is below base Monet's 75.5 and gets the smaller gain; the surviving discriminator is how broken the starting harness was.) ~~Still missing entirely: transfer to a different benchmark, which no source measures.~~ Settled 2026-08-13 for the cross-benchmark half by darwinx evolving agent harnesses: a TB2.1-evolved harness runs unchanged on SWE-bench Verified and on a different base-model family for 84.2% official pass@1, +3.4 over an LSP fix-skill reference — with the caveats that there is no matched-model base arm on SWE-V and every compared harness sits in 80.8–84.2. The same source raises the held-out gain on a huge-headroom suite to +49.5 (WebArena-Infinity, 1,260 unseen real tasks) and returns p = 0.45 on a small one, so the held-out half stays contested and the budget-matched half stays unanswered — DarwinX defines no effort tier, publishes no compute column and no dollar cost.
  • SourceIs the negative result a property of harness evolution or of Terminal-Bench? Wang et al.'s §5.2 names two conditions a fair test needs — substantial headroom above current agent scores, and a benchmark whose performance genuinely depends on the harness (specialized tools, skills, workflows) rather than one where "a shell tool and a basic prompt already suffices." Terminal-Bench satisfies neither for frontier models, and it is the benchmark essentially the whole harness-evolution literature reports on, including HarnessBank and Cline. Falsifiable directly: run the same four budget-matched arms on a harness-sensitive suite and see whether the ordering inverts. Partially answered 2026-08-13 — the second condition, not the first. darwinx evolving agent harnesses's WebArena-Infinity arm is the harness-sensitive suite by accident (a coding agent's harness pointed at Chrome), and evolution moves it 43.5% → 93.0% audit-clean on 1,260 held-out real tasks. What is still un-run is the budget-matched version of that experiment: nobody has put parallel sampling and harness evolution on the same suite at the same normalized budget where the harness genuinely binds.
  • SourceIs the Wang-et-al./DarwinX disagreement actually a search-budget disagreement? The two papers cap the search four orders of magnitude apart and neither reports it in a comparable unit: Wang et al. fix K = 5 rounds of meta-agent edits; DarwinX evolves "over many generations," screening at avg@3 on rotating subsets and confirming steering nodes at avg@5 on the full 89-task suite, and states no generation count, no rollout total and no dollar figure anywhere (the one reconstructible arm is WAI's 26 kept / 36 reverted iterations, each screened at avg@3 over 300 intents). Falsifiable two ways: cap DarwinX's generations at Wang et al.'s K and see whether the gain survives, or have both papers publish total rollouts.
  • SourceHow much of the 77.5%→88.8% survives on an agentic benchmark Cline has not been hill-climbing for six months? Every fix is described as a general harness improvement, which is a transfer claim, and no transfer measurement is reported. (HarnessBank does not touch this: it measures within-domain held-out tasks, not cross-suite transfer.)
  • SourceDoes the semantic quality-diversity archive earn its complexity against a plain greedy keep-the-best loop, holding the significance gate fixed? HarnessBank ablates the gate but never ablates the gene bank; its archive evidence is observational (credited harnesses stack mechanisms from more than one cell), not a controlled arm. Sharpened, not answered, 2026-08-13. darwinx evolving agent harnesses builds its whole contribution on the archive plus a cross-lineage merge operator and still does not ablate either — §9 says so in its own words ("the contribution of recombination relative to single-lineage mutation still requires controlled ablation"), its archive evidence is the same observational shape (four specialists solve 24/25/26/27 of 41 and the merge solves 28, on a split where one task is 2.4 points), two of those four specialists fail to beat the unevolved base, and on WebArena-Infinity every merge was reverted so the largest gain in the corpus accrued along a single lineage with the recombination machinery contributing nothing. Two papers now rest on the archive and neither has isolated it.
  • ResolvedIs there a stated test that separates narrow scaffold hill-climbing from the recursive self-improvement the term names — e.g. capability transfer to unrelated tasks, or improvement that compounds without a human re-issuing the brief? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? states the test: name what the improvement artifact is fitted to, then measure whether gains survive outside that scope. HarnessBank's two separators (cross-model transplant near-zero off the matched pathology and -15.7 wrong-way; termination at the 10-round floor vs phantom progress in 62–76% of ungated post-convergence rounds), Wang et al.'s compute-side separator (at matched budget the loop does not beat sampling five times), and Caltech's orthogonality result (a domain-fitted artifact compounds portably while the model improves not at all) instantiate it: solver-fitted gains are maintenance, domain-fitted gains are assets, and RSI would require the gains to live in the solver itself. The stated test's un-run instances remain evidence gaps: cross-suite transfer stays in this page's #oq/source items, the two-artifact controlled test on Knowledge-Centric Self-Improvement, and no source yet runs a loop that re-issues its own brief.
• Agent-Native Infrastructure2 open
  • WaitWho builds the agent-native rewrite of the long tail of human-facing services — the service owners, or a translation layer (MCP servers, computer-use agents) on top?
  • Agent-to-agent negotiation needs trust, identity, and accountability primitives that don't exist yet. What's the protocol layer, and who governs it? Partially answered: AIMS (IETF draft-klrc-aiagent-auth-03) proposes the protocol layer — agent-to-agent is just workload-to-workload, so WIMSE/SPIFFE identifiers, OAuth 2.0 delegation + token-exchange chaining, and OpenID Shared-Signals eventing (drawn from IETF/CNCF/OpenID) supply identity, delegated authority, and auditable accountability; a tool endpoint "may itself be implemented by another AI agent." But who governs it stays open: AIMS is an individual submission with no IETF WG consensus, profiling a stack of specs that are themselves still Internet-Drafts — the primitives are proposed, not ratified or arbitrated. Sharpened (2026-07): the who-governs-it answer is concretely plural — the identity/authentication/delegation slice is IETF-track (AIMS), while the authorization slice is OpenID-Foundation-track: its AuthZEN Working Group approved the AARP (a prerequisite/approval pattern generalizing CIBA — "not yet, here is what is required") and COAZ (MCP-tool-authorization) profiles as Working Group Drafts on 2026-06-15. So the protocol layer is being standardized across multiple bodies (IETF for who-you-are + how-authority-delegates; OpenID for whether-a-call-is-allowed + what-must-precede-it) — moving, but with no single arbiter and no ratified cross-body composition yet. (Standards-announcement, practitioner-opinion — proposed drafts, not settled specs.)
• Agentic Code Generation as Compilation3 open
  • SourceDoes the plan-as-typed-IR discipline transfer to a domain whose work does not decompose into a schema-declarable DAG? Every property on this page — parallel lowering, layered validation, memoized re-execution — descends from the fact that a Bridgewater analysis is a data-frame dependency graph. Falsifiable by anyone reporting the same architecture on an open-ended coding domain (refactoring, incident response) with a latency or reproducibility number.
  • SourceIs the 95% two-agent code identity a property of the typed plan or of the validation repair loop? The obvious ablation — measure identity on the pre-validation emission — separates them and is cheap for anyone running this architecture. Until it is run, "essentially a deterministic coding agent" credits the IR for work the repair loop may be doing.
  • WaitDoes the specialize-then-compound directionality claim hold — that narrow benchmarked agents compose upward more cheaply than generalists specialize downward? It is the talk's most load-bearing strategic assertion and carries no evidence.
• Automated Failure Attribution3 open
  • SourceEvery trace is one injected error into a run that otherwise succeeded, so a unique decisive step is guaranteed to exist. Does attribution accuracy survive organic failures with multiple interacting causes and no single flip-the-outcome step — where the honest answer is often "three things were marginal and the fourth tipped it"? The human panel found only 2.0% of traces had no clear decisive error, but that is a property of the construction, not of agent failures. Until this is measured the 73.9 / 25.3 numbers are an upper bound of unknown tightness. Partially answered 2026-08-13 — the collection half only, by multilingual multi agent planning failures (arXiv 2608.03735, empirical). The organic-provenance half is now sourced and the answer is that it is workable: 80 failure traces filtered by "the development agent succeeds on an English query but fails at a non-English query," real agents in a real framework on real GAIA-derived tasks, no injected faults, and a paired known-good run obtained from the language swap rather than from a warm start. The no-single-decisive-step half is untouched, and the way it is untouched is the finding: Appendix G instructs the judge that "when more than one failure occurs within a single agent plan… mark a single primary category," described as "consistent with the convention used in our manual analysis," so the multi-fault case is conceded to exist and then labelled away in both the automated and the hand-read arm. Human validation covers agreement on the assigned category (κ = 0.860) and not whether the primary pick was the sole cause. So the question narrows and gets cheaper rather than closing: re-judge those 2,689 already-labelled baseline failures with the mutual-exclusivity instruction removed and compare per-category shares — a multi-label arm over an existing corpus, no new traces needed, and the fraction of multi-fault plans falls out of it directly.
  • SourceHumans and models are never scored on the same task. Annotators ratify or correct a supplied label (94.0 / 90.0 / 90.0, κ = 0.73); models predict from scratch (73.9 / 57.5 / 22.2). So the ~20pp step gap is not a measured human ceiling, and could be much smaller or much larger. What is human from-scratch attribution accuracy on these traces? Cheaply falsifiable with the existing corpus and a blind annotation arm.
  • SourceThe base agents were chosen for unencrypted, interpretable reasoning chains because the judge must read intermediate reasoning to localize the decisive step. Frontier deployments increasingly summarize, encrypt, or discard those chains. How much attribution accuracy survives when the decisive step's reasoning is not in the trace — and is the loss concentrated in the step metric or the error-mode metric? Answerable today by re-running the benchmark with reasoning content stripped or replaced by provider summaries.
• Build for the Next Model2 open
  • SourceHow do you tell a "wait for the model" gap from a durable-harness gap before the next release? Get it wrong and you either ship vaporware or build a crutch you'll delete. Partially answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — the survivor taxonomy is the classifier: gaps in behavior/capability (task-prior structure) are "wait for the model" gaps; gaps in boundaries, org-specific record, identity, deployment structure, or human-facing legibility are durable-harness gaps that no release will close. The pre-release tell is what the fix would encode.
  • SourceDoes the strategy generalize outside frontier labs, who have privileged visibility into the next model? An external team is betting on a release it can't see.
  • ResolvedThe bet depends on a reliable release cadence and a forecastable capability curve (Task Time-Horizon Scaling). What happens to "build for the next model" if model improvement stalls (the stalled-but-diffused future)? Answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — it degrades gracefully, because correctly practiced it is a cheap call option on the release cadence: a stall costs the premium (the prototype portfolio expires unexercised), not the firm, provided market validation was kept separate from the capability bet. Three cushions: Latent Capability Overhang keeps effective capability rising post-stall (mine the model replaces wait for the model); the previously-wrong move — engineering the compensating crutch — reclassifies as correct ("too AGI-pilled" correction becomes the standing posture); and competition shifts to the durable layers that never migrated inward. The bet only fails hard under miscalibration: shipping products whose core loop depends on absent capability, which is vaporware whether or not the cadence holds.
• Claude Code Auto Mode4 open
  • SourceWhat false-positive rate does the classifier have on routine-but-aggressive refactors (e.g., large-file renames, rm of build artifacts)?
  • SourceHow well does the classifier generalize to custom tools / MCP servers where it lacks environment context?
  • SourceIs the classifier's decision boundary documented/stable enough for security-sensitive orgs to certify, or is it effectively a black box whose behavior drifts with updates? Partially answered (2026-08-04): the changelog supplies the two facts a certification argument would start from — the classifier defaults to Sonnet 5 for external sessions, and it is "validated on the session's first request and pinned for the session" (v2.1.210), so the boundary cannot move under a running agent. That closes intra-session drift and closes nothing else: no decision boundary is documented, the scope of what the classifier adjudicates widened in v2.1.218 (taking over the dangerous-rm, background-&, suspicious-Windows-path and unprovable-read-only-Bash checks from static analysis), and release-to-release drift is unaddressed and now covers more surface than before.
  • SourceDoes extending auto mode to API users change its calibration — is the classifier retrained for automation-heavy use, or held constant?
  • ResolvedCompared to OS-level sandboxing (mentioned in Claude Code Best Practices alongside auto mode), what's the defense-in-depth story? When should both be layered? Answered: Classifier Gates vs OS Sandboxing: The Defense-in-Depth Story for Auto Mode and Cowork — they are different control kinds with disjoint blind spots: the classifier is a semantic gate (judges intent the sandbox can't see — within-capability harm over allowed channels) and the sandbox is a structural barrier (contains the classifier's two documented failure modes). They fail independently by mechanism, making the stack real defense-in-depth rather than correlated friction. Layer both whenever the agent holds reach beyond the sandbox boundary (live creds, MCP to real SaaS), runs unattended, or reads untrusted input; sandbox-only is legitimate when containment is total (the Hermes Agent container-is-the-boundary design point); classifier-only is a stopgap for interactive low-stakes local work — never for unattended runs.
• Claude Code Best Practices3 open
  • SourceDoes the instruction-count ceiling hold for conditional policy, where only a handful of rules bear on any given turn? Every rule in Eliav's experiment applies to one generation simultaneously; a CLAUDE.md is mostly situational ("when editing migrations, …"), so N=80 is a floor on the harshest possible loading and says nothing about a 200-rule file of which five fire per turn. Falsifiable directly: hold the applicable subset fixed and grow the inapplicable remainder.
  • SourceHow does the Writer/Reviewer pattern compare to agent-to-agent review (as in OpenAI's Codex workflow)?
  • SourceWhen does subagent overhead exceed the benefit of context isolation? Partially answered 2026-08-03 by codex 0 to 10m users chatgpt work (practitioner-opinion, no measurement) — a task-shape criterion rather than a crossover point. Akshay Nathan (OpenAI): multi-agent modes "are best for when you have tasks that are either incredibly complicated, like open explorations, or very paralyzable… but for most tasks, they don't fall into either of those buckets," so the default should be a single agent. Note the overheads he actually names are neither context nor tokens: rate-limit consumption (Ultra "can use more of your limits," which is why OpenAI moved it behind advanced settings post-launch) and human legibility (sub-agent transcripts hidden by default to avoid overwhelming users — see Shared Harness, Differentiated Surfaces). A practitioner counter-practice in the same episode pulls the other way: Vibhu reports telling every long-running task to "use sub-agents where possible" for wall-clock and for cost, fanning out to cheaper models — which Cost-per-Task Over Cost-per-Token argues is the wrong default. Second instance (2026-08-04): Willison runs the same practice with the tier choice itself delegated — "for all coding tasks use your judgement to decide an appropriate lower power model and run that in a subagent" — and reports only that his Fable allowance shrinks more slowly. Two practitioners now default to the fan-out; neither measures it. What remains open is the measured crossover, which no source in the corpus supplies.
  • ResolvedWhat's the optimal CLAUDE.md length before instructions start getting lost? Is there a measurable threshold? Answered 2026-08-04 by prompt design at scale (Eliav, arXiv 2607.19257, empirical) — full treatment at Instruction Compounding. Yes, there is a threshold, and it is count-shaped rather than length-shaped: across five models (including Claude Sonnet 5 and Haiku 4.5), the rate at which every instruction in the prompt is obeyed falls steeply by N≈40 simultaneous verifiable rules and hits zero by N≈80, holding through N=160 and identical across markdown, plain text, prose, and table renderings and across system-prompt vs. user-turn placement. The paper's own prescription is the answer in usable form: 40 simultaneous instructions is a redesign point, not a tuning point — past it, splitting across turns, tools, or a validation pass is the only thing that works, and reformatting is not. This also settles the residual the 2026-08-03 retag left behind (does aggregate size have an independent effect once each line passes ablation?): yes — every rule tested was distinct, non-redundant, and individually satisfiable, so per-line ablation non-inferiority would have cleared all of them and still missed the collapse. Two scope limits carried on the answer: "perfect response" is a strict conjunction, so some of the floor is the arithmetic of ANDing N checks rather than the model dropping the block, and every rule tested is a hard output constraint applied to a single generation. The conditional-policy case a real CLAUDE.md actually presents is now the successor question in Open Questions above.
• Client-Side Agent Optimization5 open
  • SourceHow does combination-level optimization interact with continual model releases? If Claude Opus 4.7 ships next month, does the full Pareto frontier need re-running, or do warm-started bandits adapt cheaply? Partially answered (2026-08-04, by synthesis): What Makes a Self-Improvement Artifact Transfer? — a combination is a solver-fitted artifact (fitted to the current menu's capabilities and prices), so the frame predicts full re-runs rather than cheap adaptation, with the durable residue being assignment rules rather than assignments; the HotpotQA→Cursor inversion below already shows the assignment not transferring while the reconciling rule does. The warm-started-bandit half is an empirical question no source measures.
  • SourceAt what pipeline depth does the combinatorial search become intractable even for Arm Elimination? The paper tests up to ~81 combinations; production pipelines with 5+ roles and 10+ candidate models each blow past that.
  • SourceDoes the "weak planner + strong solver" pattern generalize, or is it specific to HotpotQA's delegation dynamic? Recommender-critic, drafter-editor, and retriever-generator topologies might invert. Partially answered — it inverts (2026-08-03): on Cursor's long-horizon build task the efficient frontier is the opposite assignment, strong planner + cheap worker, with the entire worker fleet costing $411 under an Opus 4.8 planner versus $9,373 when a frontier model did both jobs at the same quality. The reconciling variable is what the planner is able to do: HotpotQA's planner could answer directly and did; Cursor's cannot. So the pattern is specific to the delegation dynamic, and the general rule is about foreclosing execution at the planner role rather than about weakening the model in it.
  • SourceWhat's the right way to re-evaluate when the tool environment changes? AgentOpt assumes fixed tools — adding or removing a tool potentially invalidates the whole frontier.
  • SourceIs there a cheap per-call classifier that can predict which combination will win on a given query, avoiding combo-level evaluation entirely? Sharpened (2026-08-03): Writer's harness swap proposes classifying on feature demand rather than difficulty — which orchestration features (delegation, MCP tool use, multi-step workflows) a request will exercise — on the evidence that those features carry capability floors and that a model below the floor fails on them regardless of how simple the prompt reads. That is a candidate classifier target, not a classifier: nobody has built or evaluated one.
• Codex App Server Protocol3 open
  • SourceIs there a public schema registry so external orchestrators can target specific App Server versions without generate-json-schema?
  • SourceThe "dynamic tool calls (experimental)" caveat — what's the stability roadmap? Symphony depends on this for its security model.
  • SourceHow well does the protocol handle multi-modal turns (image inputs, screenshot attachments)? The spec is text-focused.
  • ResolvedHow does the App Server protocol compare in detail to MCP? Both expose tools to a model, but App Server is inside the Codex runtime while MCP is outside. When does each win? Answered: App Server vs MCP, and the Claude-Side Equivalent: Three Boundaries for Driving Agents — they sit at different planes and mostly compose rather than compete: MCP is the model↔world tool plane (write once, consume on every surface, provider-operated), App Server is the orchestrator↔runtime session plane (thread lifecycle, turns, events, timeouts, token accounting — none of it in MCP's scope). The only overlap is dynamic tool calls, where the rule is: MCP for reusable cross-surface third-party capabilities; orchestrator-injected tools for session-scoped, credential-sensitive ones (the linear_graphql token-isolation pattern, which also shrinks the poisoned-metadata/rug-pull attack surface to first-party code) — at the cost of experimental stability and zero ecosystem reuse.
  • ResolvedIs there an analogous protocol on the Claude side, or is Claude's equivalent exclusively the Agent SDK + tool-use API? Comparing the two would clarify when "drive an existing CLI" beats "build on the SDK." Answered: App Server vs MCP, and the Claude-Side Equivalent: Three Boundaries for Driving Agents — no documented Claude-side protocol; the offering brackets the App Server's position: claude -p (drive the product, inherit the full harness — permissions with abort-don't-hang unattended behavior, skills, context files, MCP wiring — but get text, not structured events) and the Agent SDK (build a different product on the raw runtime — Claude Design's weekend prototype). Rule: drive the CLI when the product's harness is the value and orchestration is batch/fan-out shaped; build on the SDK when the agent is a different product with its own surface and tools. Symphony's own tmux→protocol evolution marks the middle layer (structured session control over the product harness) that the Claude side currently approximates from either end — whether it gets standardized or harness shrinkage makes it moot is the watch item.
• Context Lifecycle Management6 open
  • SourceDoes the input-token reduction survive a matched billed-cost audit once side-channel planner calls and prefix-cache breaks are charged? The paper reports prompt-surface impact and explicitly declines this claim. Partially answered (2026-08-03): Prompt-Cache Economics supplies the audit for the class but not for Self-GC — CAPC reconciles measured API spend against Anthropic's invoice to within 1% and finds a token-reducing technique (query-aware compression, 3× fewer tokens) costing +40.1% more than sending nothing compressed on τ-bench retail. So the concern the question encodes is real and measured; a Self-GC-specific billed-cost audit still doesn't exist. Annotated (2026-08-04): agentic context management supplies an analytic bound on one half and explicitly declines the other. The overhead half: because each compaction pass operates on the already-compacted context plus recent turns, the number of passes grows only linearly and total cost is N·W·(1 + c/p) — a fixed multiplicative factor rather than a growing tax (~1.25× at the paper's illustrative p = 8, c = 2). If Self-GC's side-channel planner call behaves the same way, planner overhead cannot asymptotically eat the savings. The cache half is dropped by assumption ("ignore caching discounts, which shift the constants but not the asymptotics"), which is precisely the term the question is about — so this narrows the question to the cache break alone and answers none of it with a measurement.
  • NowIs the 0.3 expected-pruning break-even for immediate commit portable, or a function of one provider's cache pricing and TTL? Stated as an operating policy over one deployment's regression. Partially answered (2026-08-03): Prompt-Cache Economics settles the portability question in principle — the analogous threshold is ρ_cross(r) = (α − 1/r)/(α − β) over the write premium and read discount alone, and it moves sharply with both provider and TTL (α = 1.25 on Anthropic's 5-minute cache vs 2.0 on its 1-hour cache; OpenAI α = 1.0, β = 0.5). Read across, the number is not portable and the derivation is. Re-deriving 0.3 under that parametrization is now a synthesis over pages already in the wiki.
  • SourceHow does object-level GC compare against clear-and-restart on the same traces? No source in the corpus measures the two against each other, and they optimize different things — GC preserves in-run dependencies, clearing resets attention quality.
  • SourceHas anyone other than a vendor measured that validated compaction actually preserves fidelity where crude summarization does not? The whole three-regime argument turns on a cell — linear cost with checked fidelity — that no source in the corpus has isolated. Maximem's 92.0%/93.2% are end-to-end system scores on conversational-memory benchmarks with no compaction ablation, its validation mechanism is undisclosed, and Self-GC's no-impact judge grades candidate plans offline rather than a returned validation score. The falsifiable form: an A/B of the same compactor with the information-loss check on and off, on the same traces, scored on downstream task success.
  • SourceDoes the "no semantic interference from coordination volume" null survive at the prompt lengths agents actually run? RCWT's intact-task ablation holds at ceiling to a 0.95 coordination ratio, but its largest condition is ~14,000 total tokens — while the sessions this page's other sources measure average 70–90k input tokens per request. The falsifiable form: rerun the intact-task ablation with the same 698-token task block at 100k, 250k and 500k of surrounding coordination content, on models whose per-model effective ceilings Context Window Smart Zone shows are not predicted by the advertised window. If the null holds there, displacement is the whole story; if it breaks, there are two mechanisms and the corpus has been attributing both to one.
  • SourceWhat fraction of agent memory failures are reasoning-sufficiency failures rather than retrieval failures — the bridge document missing while a relevant document was returned? Every memory benchmark in the corpus scores a hit against a single gold target, so the quantity is structurally unmeasurable by all of them; the only bearing datum is that LongMemEval's multi-session category (75.2%) carries nearly all of one system's residual error. Named trigger: Maximem states a benchmark measuring accuracy, latency, token efficiency and context-rot resistance together is forthcoming.
• Context Window Smart Zone3 open
  • SourceDoes the smart-zone marker scale with model size, or is it bounded by attention architecture? Pocock observes "the dumb zone has become less dumb lately" but pegs it at 100K through 2026. Partially answered 2026-08-04 by prompt design at scale (empirical) — neither cleanly, because the premise of a single marker doesn't hold. Degradation onset is a per-model effective ceiling that the advertised window does not predict: two models sharing the same documented 1,000,000-token ceiling diverge sharply in format-spread growth over the identical 256k→512k range. It also splits the question by task — retrieval holds at 0.98–1.00 through 64k and past 128k for some models, well beyond the 100K marker, which is consistent with Pocock's own retrieval-vs-reasoning carve-out. The paper measures no reasoning task, so whether the ~100K reasoning marker is architectural remains untested.
  • SourceWhen sparse-attention or memory-augmented architectures ship, does the smart zone become a soft constraint? Partially answered (2026-08-12) — the trigger event landed, and the first evidence points the other way. Sparse attention is now, in Ren et al.'s words, "widely deployed in long-context serving stacks" — so this is no longer a prediction. Their dense-calibrated audit (empirical) supplies the mechanism that argues against the soft-constraint hope: block selection does not merely make more context affordable, it severs cross-block attention, and an ablation isolating one probe block from cross-block communication collapses its behavioural influence from 4.48 logits to exactly zero across all 1,536 units, with sparse partial connectivity sitting between the two. Since the smart zone is a claim about reasoning over the window rather than retrieval within it, and reasoning is precisely what needs blocks to talk to each other, a cheap window bought by discarding blocks is not the same object as a large window. Three reasons this is only partial: the outcome is a logit-margin influence proxy with no reasoning or task-accuracy measurement anywhere; all models are 7B–8B; and the audit never varies context length, so it says nothing about where a sparse model's own effective ceiling sits. Retagged out of the Predictions ledger now that the architectures have shipped.
  • SourceHow should harnesses surface remaining smart-zone budget to the user — token count, percentage, or a richer signal?
• Cost-per-Task Over Cost-per-Token3 open
  • SourceDoes "cost-per-task is lower for more intelligent models" survive measurement on non-Anthropic production traffic? The claim is stated without data and the published curves are explicitly illustrative. Still open, with a near-miss (2026-07-30): DeepMind's Gemini 3.5 Flash-Lite card is a non-Anthropic instance of the shape — +67% output price, a full agentic tier of capability — but reports no tokens-per-task, so it supplies the premise and not the measurement. Partially answered (2026-08-03), and it splits: Cursor's four model mixes are the measurement — non-Anthropic infrastructure, a real four-hour workload, matched time budgets, matched quality, published dollars. Within the planner role the claim holds: the more expensive Fable 5 planner billed slightly less than Opus 4.8 at roughly twice the per-token price, because it emitted far fewer planning tokens. At the level of the whole run it fails: the same Fable configuration came out substantially more expensive because its workers burned several times the tokens, and the most expensive run of all was the strongest model used throughout ($10,565 versus $1,339). So the thesis appears to be a claim about a role, not about a system, and no source yet measures it on a single-agent workload outside Anthropic. Sharpened, with the first counter-datum (2026-08-03): Writer's harness swap publishes per-model cost and per-model quality for six models on non-Anthropic infrastructure under one pinned price table, and cost per task rises monotonically with model strength on that workload — quality per dollar is worst for the two strongest models (Palmyra X6 3.16, Sonnet 4.6 3.27) and best for the cheapest (Qwen 3.6 4.44). It is a controlled bench rather than production traffic, the arms are not iso-quality, and the capability spread is only eight points, so it does not close the question — but the sign is wrong for the vendor guidance and the paper's own conclusion is that the model menu is the smaller lever anyway. Closest yet, and it splits again (2026-08-04): Databricks' internal coding bench — real engineering tasks on its own multi-million-line codebase, measured by neither Anthropic nor a model vendor — puts Opus 4.8 at $1.94/task and 87% success against Sonnet 5 at $2.09 and 81%, on tokens ~1.7× cheaper. Within Anthropic's own line the claim therefore holds, in the long-horizon coding regime where its mechanism should be strongest, and this is the first time it holds on a third party's real codebase. Against the wider menu it fails: open-weight GLM 5.2 is statistically tied with Opus 4.8 on quality at $1.28/task. So the surviving form of the rule is about tokens-to-completion, not about price tier — a cheaper model is dearer per task when it burns more tokens and finishes less often, which is contingent, not structural. Still not closed: case-study secondary reporting, production-derived tasks rather than production traffic, and no n, variance or per-arm methodology behind "statistically tied." The article's own cited generalization (arXiv 2603.23971 — a third of comparisons invert; Gemini 3 Flash 80% cheaper listed, 38% dearer in practice) is the paper most likely to settle this and is not yet ingested.
  • SourceDoes the advisor strategy's result (within 10% of the advisor's score at 63% of its price) generalize beyond SWE-bench Pro and the Sonnet-5/Fable-5 pairing — and where is the crossover at which advisor calls cost more than they save?
  • SourceIs "start with the strongest model" safe inside multi-role pipelines, given AgentOpt's finding that the strongest model was the worst planner? Anthropic's Sonnet-for-sub-agents note hints at a boundary it never states. Partially answered (2026-08-03): Cursor's production swarm says the safe form of the rule is positional — strongest model as planner, cheapest capable model as worker — and that running the strongest model in every role is the single most expensive way to reach the same grade. It also dissolves the apparent conflict with AgentOpt: Opus was the worst HotpotQA planner because it answered from parametric knowledge instead of delegating, and Cursor's architecture makes that impossible ("a planner never implements"). The failure is a property of harnesses that let a planner execute, not of strong models in the planner seat. Still unsettled: whether the ordering survives on tasks where the worker's job is judgment-heavy rather than instruction-following, which is the regime Cursor's own framing exempts.
• Crystallizing Agent Work into Workflows2 open
  • SourceDoes the lifecycle transfer out of IT operations? Every number here comes from incident response, which is unusually repetitive and has a crisp success oracle (the incident resolved). Coding, product and research work have neither property in the same degree. The falsifiable version: apply the Type 3→2→1 promotion criteria to an agentic coding pipeline and measure whether the deterministic share rises at all over comparable time. Checked, not answered (2026-08-13): shopify sidekick continual learning loop was scouted for this and fails it on three counts. (1) Wrong domain shape, not a different one. A merchant-facing agent that introspects a GraphQL schema, looks up filter syntax and executes a query against the Admin API is structured-query generation against a machine-checkable target — closer to incident response than to a feature request, and it shares exactly the two properties this bullet says coding lacks: the task recurs across millions of merchants, and correctness is largely oracle-checkable because the query either returns the right rows or it does not. It is the same shape in different clothes. (2) No transfer claim is made. The piece is scoped to one agent, and the ingest pass found no assertion anywhere that the loop generalizes to other Shopify domains, let alone out of them. (3) It is not this lifecycle. Nothing is promoted toward determinism — the endpoint is model weights, the runtime stays stochastic, and there is no demotion mechanism at all (see the third-endpoint section above). A single-agent vendor account cannot carry this question in either direction, and marking it answered on one would be the error this annotation exists to prevent. The falsifiable version stands unchanged.
  • SourceThe platform-level cost curve has no counterfactual arm. How much of the >70% per-incident cost fall is crystallization, and how much is ordinary model-price decline plus caching over the same eight months? A controlled comparison, or a decomposition against contemporaneous list prices, would separate them.
  • ResolvedCrystallization and harness shrinkage give opposite instructions at a model upgrade — delete the scaffolding versus keep the evidence-gated permissions. Which governs, and does the answer differ for instruction scaffolding versus authority scaffolding? Answered: Authority and Audit Survive Abundance — neither governs the other; they govern disjoint objects, separable by one test: can the model being better make this line unnecessary? Instruction scaffolding encodes a task prior (shrinkage governs — ablate at every release); this page's authority scaffolding encodes a boundary plus a local evidence record, neither of which a capability jump supplies — and the security corpus makes the stronger claim that authority cannot migrate inward, because a component that grants its own scope is circular ("you can delegate judgment; you cannot delegate authorization"). The sort is by what a line encodes, not where it lives (a prompt-borne scope declaration is still authority-class; the derived page grounds this in the constraint/request asymmetry). At upgrade day the reconciliation is already in this page's design: the launch pass prunes instructions, permission grants stay untouched, and the demotion circuit-breaker re-earns authority from evidence continuously — so the upgrade moment requires no authority decision at all.
• Deep Modules for Agents3 open
  • SourceHow big is "deep enough"? Pocock's example modules are several hundred LOC; Ousterhout's textbook examples are larger. There's a sweet spot; not articulated.
  • SourceFor ports/adapters codebases, does the deep-module advice transfer cleanly? The "small interface" is the port; the "large behavior" is the adapter. Probably yes, but not exercised in source.
  • SourceRefactor cost vs benefit: when is "improve-code-base-architecture" worth running on a working repo?
• Deep Research Agents5 open
  • WaitDoes the orchestration advantage shrink as base models cross the next thresholds, or is open-ended retrieval/synthesis a durable harness asset (unlike, say, prompt scaffolding)?
  • SourceDRACO grades single-turn interactions only. How much of real deep-research value is in the multi-turn loop (clarifying questions, follow-ups) that the benchmark doesn't yet measure?
  • SourceFactual accuracy is the weak axis everywhere — is the fix better retrieval, better verification-in-the-loop, or a tool-grounded check the way Lean grounds proof search? Partially answered by is deep research reliable (arXiv 2607.20891, empirical), which rules out one branch and measures a second. Not retrieval: misleading-evidence reach is already 72–98%, search-result rank moves FCAR by 1.7pp, and adding documents past the first buys nothing — retrieval is not the filter and making it better cannot be the lever. Verification-in-the-loop helps and does not suffice: pre-research verification prompting takes DeerFlow from a 60–76% baseline to 37–57%, a post-research refinement agent to 20–58%, the combination to 15–62% — and the combination is worse than either alone for Intern-S1-Pro, because the refinement step re-retrieves from the same poisoned pool. The tool-grounded branch is untouched and remains the open half: open-domain factual claims have no Lean, and the nearest thing measured here (a search-enabled verifier) is exactly what already gets these documents right in isolation and is never invoked in-workflow. So the live question narrows to where in the workflow verification must sit, not whether it helps.
  • SourceDoes any context-pruning policy sit on the Pareto frontier of all three graded axes — rubric quality, citation recall, and ground-truth key-point coverage — against an unpruned run? Across 39 configurations in marginal value estimation deep research none does: the best pruned KPR+KPC is 66.32 against 70.23, and the three arms that hold citation recall give up half their relevance. Either compression on this workload has an irreducible coverage cost, or nobody has yet built a pruner whose objective is coverage rather than relevance-minus-redundancy — the falsifiable form is a pruning rule scored directly against retained key-point recall, which none of the eight rules here is.
  • SourceThe verification asymmetry is inferred, not isolated: the misleading corpus was selected for unanimous verifier agreement, and the standalone verifier's job (judge one document, with search tools and explicit search discipline) is strictly easier than the agent's (judge a document while executing a research task). Does an in-workflow verification step given the same tools, the same focused prompt, and its own budget close the gap — or does carrying a task degrade the check regardless of how it is prompted? The pre-research defense is the weak version of this experiment (it asks, but grants no separate step and no tools) and recovers roughly half the gap; the strong version has not been run.
• Deterministic Engineering for Agent Code Review3 open
  • SourceThe paper has no ablation of any kind, so which of the three deterministic injections carries the 2.17×, and whether any of it survives an instruction-only baseline, is untested — the same gap reason less verify more leaves in its own domain. The falsifiable form is cheap and the code is open-sourced: run OpenCodeReview with reflection disabled, with rule-guided dispatch replaced by agent-driven triage, and with the six bounded tools swapped for a shell; then run Claude Code with OpenCodeReview's built-in rule text pasted into its prompt. Until at least the last of those exists, "determinism beats autonomy" is a design philosophy that has twice been measured against a confound rather than against its own alternative.
  • SourceIs 20% recall a ceiling of the constrained design or of the review task? Across all twelve configurations no system reaches both precision above 25% and recall above 25%, and within OpenCodeReview the best-precision backend is tied-worst on recall — so the empty upper-right quadrant of Figure 3 may be an architectural limit or may be the benchmark's. Distinguishable without new methods: sweep OpenCodeReview's reflector threshold and its 30-iteration bound to trace out its own precision–recall curve, and report whether the curve bends toward the empty quadrant or slides along the same frontier the baselines sit on.
  • SourceEvery figure here is a Qwen3-235B-A22B-Instruct match verdict with no published agreement against human matching judgments, on a benchmark whose labels were human-validated to three rounds. Does the matcher agree with expert adjudication well enough to support 12-way ordering, and how much of the five-run mean's stability is genuine rather than the ρ ≈ 0.66–0.97 dependence a homogeneous jury carries? Both are answerable on a few hundred sampled match decisions plus the per-run spread the authors already have and did not print.
• Deterministic Pre-Execution Gates3 open
  • SourceThe paper never runs the obvious baseline: does forcefully prompting the four gated rules, or adding a reflection step, recover the +12.4pp on the non-deceptive failures? The authors argue prompting cannot help where the user asserts false state, which concedes the rest. Until someone runs it, "reason less, verify more" is a claim about the deceptive slice generalized to the whole.
  • SourceHow much of the recovery is the block and how much is the rejection message? A gate returns a structured reason the agent re-plans on, so the lift may be partly informative feedback rather than prevented corruption. Falsifiable cheaply: rerun the suite returning a generic "rejected" with no reason and compare. The deterministic guarantee over the blocked write is unaffected either way; the attribution of the 12.4pp is not. Partially answered (2026-08-04) by Harness-Induced Belief Divergence (Yi & Song, arXiv 2607.04528, empirical), from the safety side rather than the success side: a block that withholds its reason leaves the disposition intact — 42 of 60 blocked destructive-command steps re-propose a same-class risky action within three steps (UnsafeRetryRate 0.700) — and gating measurably relocates the model's failure attribution onto the harness policy. That is evidence the message carries real weight, but it is not the requested attribution: that paper reports no task-success number at all, and never runs the with-reason vs without-reason contrast. The cheap experiment is still unrun.
  • SourceGate precision was audited against ground-truth trajectories, which a deployment does not have — and without that audit baggage_allowance (5% precision, 40 false blocks in 42 fires) ships silently. What deployment-time signal substitutes: post-rejection completion rate, human adjudication of a rejection sample, or a per-gate A/B? Nothing in the corpus proposes one, and a gate suite with no precision signal is a new silent failure mode wearing the old one's clothes. Partially answered (2026-08-12) by Structural Artifact Monitoring (Ravindra, Tiwari & Wolowski, arXiv 2607.14570, empirical), which runs a procedure needing no attack labels at all: calibrate the threshold on honest episodes only (n = 100, pooled across five honest replicates) to a stated false-positive budget, freeze it before any attack evaluation, and then report honest-task completion under the live monitor as the utility check — 81/100 against a no-intervention 82/100. That is the corpus's first worked instance of this question's own first candidate, post-rejection completion rate, and it uses nothing a deployment lacks: honest traffic and a task outcome. Two things keep it partial. It is a monitor-level signal — it sizes the aggregate false-block cost and cannot attribute a false block to one predicate, so a baggage_allowance-shaped gate would still ship invisibly inside a suite whose aggregate FPR looked acceptable. And that paper's own arms expose the measurement floor: its git-diff rollback arm scores 5 points above the no-intervention ceiling, which a blocking monitor cannot cause, so an unpaired n = 100 resolves nothing smaller than roughly 10 points.
• Document Parsing as the Retrieval Bottleneck2 open
  • SourceDoes the "parsing errors propagate into 7 of 12 pain points" cascade hold up under measurement rather than assertion? Layerwise Omission Attribution supplies the instrument — canary taps make an L0 loss exactly countable, and the (conflict − literal) needle contrast isolates downstream behavioral loss — so the falsifiable version is: on a fixed corpus, parse with a structure-preserving and a flat-text parser, hold every later stage constant, and attribute the delta in end-to-end failures by layer. Nothing in the corpus does this.
  • WaitHow far has the specialist-parser advantage over general frontier VLMs actually narrowed? ParseBench Fig. 5 (a vendor-run benchmark) puts LlamaParse Agentic at 84.9% against Gemini at its high setting near 76%, at roughly half the cost per page. If Harness Shrinkage as Models Improve governs here as elsewhere, the specialist layer is a temporary tax on current VLM weakness; if bbox grounding and per-page cost predictability are structural, it is not. Falsifiable at the next frontier VLM release by re-running the same harness.
  • ResolvedIs the audit-trail argument for retrieval strong enough to survive genuinely cheap long context? The deck's three reasons are cost, governance and auditability, and only cost is a function of token price. If a 1M-token window becomes ~free, does per-chunk permission filtering and citation-log auditability still force a retrieval layer — or do they become an attribution problem solvable inside the window? Answered: Authority and Audit Survive Abundance — yes, wherever the requirements it serves exist, because only the cost leg is token-priced. Governance survives by circularity: per-chunk permission filtering is per-call authorization at the retrieval boundary, and enforcement cannot live inside the window it polices ("'model promised to ignore' is not a boundary" is the in-band collapse the security corpus measured) — the window is the wrong trust domain at any price. Auditability survives because in-window attribution is model testimony where an audit needs a log produced outside the model — and a log without selection reads "everything," which attributes nothing: retrieval is the act that makes a citation log non-trivial. The scope conditions run the other way too: governance forces a retrieval layer only where principals > 1, audit only where accountability is required, and the leg this page's "~free" premise prices too generously is capability (Context Window Smart Zone's effective-ceiling/refusal evidence) — the one leg made of current model limitation rather than structure. The requirement also binds the compiled-wiki rival: compilation and retrieval are both selection-with-a-record, and corpus-stuffing is the only architecture the audit leg eliminates outright.
• Dynamic Workflows: An Algebra for Agents4 open
  • SourceThe "algebra" is still unpublished as a vocabulary. Sumner publishes the loop's shape (pop a task → implement → parallel review → apply) and its inventory (~50 loops), but no combinator names, no composition operators, and no workflow source. Partially answered: the sequence/parallel primitives are visibly in use and the model demonstrably authors and edits them from English instructions mid-run; what's missing is whether there is any structure beyond while + Promise.all. Annotated 2026-08-04: the changelog exposes the first product-surface handles on a workflow — a size guideline (workflowSizeGuideline, small/medium/large, default "fewer than 15 agents"), workflow.run_id/workflow.name OTel attributes, an agent grid, and a running-workflow status line — but still no combinator names and no workflow source. Size and observability, not vocabulary.
  • SourceIs model-authored orchestration more token-efficient than a hand-built harness for the same task? Partially answered — one side now has a number — 5.9B uncached input / 690M output / 72B cached reads / ~$165k for the Bun port — and Sumner's remark that "I would've had to write my own harness to pull this off otherwise" concedes the comparison was never run. No counterfactual harness exists, so the efficiency question is unanswerable from this source and needs a task run both ways. Complicated further (2026-07-27): the ~$165k is token cost to the merge only, excluding CI, employee time, and an ongoing post-merge tail, so even the one number on the board is not the campaign's cost. Sharpened (2026-08-03): the comparison the question asks for may be against the wrong baseline. OrchBench prices orchestration against a single serial agent rather than against a rival harness, and finds multi-agent plans consume roughly 1.5× the tokens of serial execution across every planner and context limit it tested — agent startup (1,200 tokens each), cross-agent communication, and compression overhead are unavoidable costs of fanning out at all. If that holds outside simulation, the honest form of this question is not "is model-authored orchestration cheaper" but "what does the token premium buy," and OrchBench's answer is: quality only while the working state overflows one context window, plus wall clock. The experimental design now exists, run on the wrong pair (2026-08-03): Cursor re-ran the same task under two harnesses at fixed models and a fixed time budget — the exact controlled shape this question needs — and the deliberately engineered harness reached the same grade with a fraction of the commits, conflicts and code. But both arms are hand-built, so it prices harness engineering, not model authorship. Someone now has to run this design with one model-authored arm.
  • WaitWhat did the Bun port cost to shipped rather than to green — CI, employee time, and post-merge agent spend included? Lockwood guesses ~$800k from an assumed $10k/day, which is not an observation; settling it needs either a first-party total or a public v1.4.0 release that closes the tail and dates it. The related tell is whether the ~2,475-and-rising open robobun PR queue drains (stabilization debt) or holds steady (continuous-agent-fleet throughput).
  • SourceHow far does the pattern degrade without a verification substrate? Sharpened, not answered: Bun's oracle had a property most codebases lack — the test suite was written in a different language from the implementation, so it survived the port unchanged; assertion count (1M+) is the visible variable but language-independence is the load-bearing one. What would settle it is a comparable port where the tests are written in the source language. Corroborated, still not answered (2026-08-03): Cursor's SQLite swarm is the second giant-swarm success in the corpus and its oracle has the same property in a stronger form — sqllogictest grades query results across different engines, so it is independent of the implementation entirely, not merely of its language. Two-for-two on implementation-independent oracles moves the confounder from "possibly incidental" to "possibly necessary" and supplies no negative case. The question is unchanged and now better motivated.
• Failures That Look Like Success2 open
  • SourceIs "internal state correct, final message stale" a general LLM-agent failure signature (state/utterance divergence) or an artifact of session-state architectures like ADK's? A cross-framework tally would tell.
  • SourceWhat fraction of production agent failures are silent-contract violations vs. loud errors? The 14/15 and 3/4 numbers are demo-sized; telemetry-scale data (Production-Sourced Evaluation) could ground the class. Partially answered, and reframed, 2026-08-03 by reason less verify more (empirical): 78% of observed failures on the τ²-bench airline domain are silent wrong-state failures with no tool error — benchmark-scale (250 trials over 50 tasks, replicated on 15 disjoint seeds), not production telemetry, so the question's original ask stands. What it does settle is that the question is mis-posed as a single number: the same paper's negative controls find no silent class in τ²-bench retail (self-enforcing tools raise loud errors) and zero gate firings across 200 BFCL entries. The fraction is set by whether the tool layer enforces its own preconditions, so the answerable version is "what fraction of production tool surfaces are policy-permissive?" — still unmeasured, and now the cheaper question. Explicitly not advanced 2026-08-03 by where facts go missing, despite carrying larger numbers: its 75,476-trial waterfall is a fault-injection decomposition the paper refuses four separate times to read as prevalence, and its 372-trial real-data pilot scores end-to-end non-success (0.578, or 0.509 excluding 52 execution errors) on an endpoint broader than silent omission. It does independently name a representative deployment study as the missing work — two labs now pointing at the same gap from opposite ends of the pipeline. Also explicitly not advanced 2026-08-04 by who and when pro — the corpus most likely to be mistaken for an answer, at 12,326 failed trajectories across 26 benchmarks. It cannot advance the question by construction: every trace is a deliberately injected error into a run that had already succeeded, so its failure distribution is designed rather than observed, and no trace in it was sampled from anything running in production. Partially answered 2026-08-04 — and this is the telemetry the question asked for. Transluce judged 8,600 unmodified real coding-agent sessions and found overselling present in 34.7% of public SWE-chat sessions (severe in 1.8%) and monitor evasion in 14.7% (severe in 1.9%). Overselling is this page's signature — a confident report of work that was not done — and one session in three containing one is the first production-scale number attached to the class. Three things keep it partial. The denominator is sessions, not failures, so it does not answer "what fraction of failures are silent"; a session can contain an overselling claim and still succeed. Omission was deliberately excluded — Transluce says catching "lying by omission" required reading whole transcripts and chose not to measure laziness or neglect for the same reason, so the read-side half of this page is untouched. And the rate is confounded by the user's own process: monitor evasion cannot exist in a session with no gates, and one heavy user supplies 41 of 76 charted severe cases.
• Harness Build-vs-Buy3 open
  • SourceWhat fraction of upstream churn does a narrow fork actually inherit? 13 PRs/day is the full-surface upper bound; nobody has measured the delta for a fork that drops the UI, multi-provider support, and deployment surface. A rerun restricting the diff to a realistic kept-file set would settle it.
  • WaitDoes the merged-PR rate of coding-agent harnesses peak and fall as models improve, or keep rising? Harness Shrinkage as Models Improve implies an eventual peak; the observed trajectory (Codex ~124 → ~1,000 PRs/month over 12 months) is still climbing. Re-pull the same four repos in twelve months.
  • SourceIs bug-fix share comparable across projects at all, or is the 16–68% spread purely label hygiene? Shah says compare directionally and flags Hermes' 68% himself; a uniform commit-message classifier rerun across the four repos would separate signal from convention.
• Harness Shrinkage as Models Improve2 open
  • WaitThe Boris "100 lines" prediction is a year out from May 2026 — testable in 2027. Partially answered: Harness Build-vs-Buy supplies the first measurement of harness codebase size and its trend — four comparable harnesses at 1.05M–1.75M lines with merged-PR rates still climbing (Codex ~124 → ~1,000/month over twelve months). That is evidence about the reference class, not about Claude Code, which is closed-source and unmeasured; the 2027 test stands.
  • WaitIf harness work shrinks, what new work expands to fill it? Cat Wu's bet: PM/product taste, eval-writing, character work.
  • ResolvedDoes all prompt scaffolding eventually migrate into the model, or does some remain — e.g. organization-specific style, security rules, brand voice? Answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — no: only behavior requests migrate (and past expiry turn harmful per Instruction Compounding). Five classes survive, sorted by the bitter-lesson exemption rule (structure encoding a task prior migrates; structure encoding boundaries, records, identity, or serving arithmetic doesn't): boundary enforcement (security rules survive as constraints, the instruction class that keeps working), organization-specific record (a smarter model can't infer an unrecorded decision — org style qua arbitrary convention lives here), deliberate identity (brand voice/character is held stable across capability jumps by design), inference/deployment structure, and human-facing legibility (which grows). Communication calibration flows the opposite way — added, not removed, as defaults lengthen. "Harness shrinkage" is really request shrinkage; the prompt converges to a residue of constraints, records, identity, and calibration.
• Harness-Induced Belief Divergence3 open
  • SourceDoes harness-induced belief divergence actually cost anything? The paper's framing claim — divergence at preserved terminal success — is never measured, and the falsifiable version is cheap: report pass rate per harness alongside D_growth on the same grid. Until someone does, "the harness changes beliefs but not outcomes" and "the harness changes beliefs and outcomes and this paper cannot see which" are equally consistent with every number here.
  • SourceHow much of D_arrival is a real interface difference and how much is constraint-string vocabulary? Lemma 1 guarantees D_belief >= 0.25 from disjoint constraint sets alone, Assumption 1's canonical embedding is supposed to remove exactly that, and the observed 0.975–1.000 arrival readout says it does not. A semantic constraint matcher (embedding or entailment) in place of normalised-string Jaccard would separate the two in one re-run.
  • SourceDoes the UnsafeRetryRate result generalize past the 0.700 measured on one 15-task group with one unnamed model — and does making the block's reason visible reduce it? The paper argues blocked-action logging should, but measures logging's effect on divergence, never on retry rate. This is the same question Deterministic Pre-Execution Gates asks about the rejection message, from the safety side rather than the success side.
• Instruction Compounding3 open
  • SourceIs the review case the same mechanism as over-verification, or two? "Be conservative" reducing detection looks like high-fidelity literal instruction-following; over-verification looks like behavioral summation. Distinguishable by testing whether a weaker verification instruction still over-verifies.
  • NowAnthropic's list is hand-curated per release. Is there a detectable signal — from eval deltas, token counts, or the model's own read of its system prompt — that flags which existing prompt lines have become compounding, so pruning is not a manual reread? Partially answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — the signature exists: a compounding line shows ablation non-inferiority (removal holds or improves quality while cutting tokens — Anthropic's own criterion, and the decisive per-line test) and inverted dose-response (escalating the instruction worsens the metric — the Unproductive Self-Verification effort-inversion fingerprint, detectable from eval deltas without ablation). Native-behavior baselining (does the uninstructed model already do it?) ranks candidates, and only request-form lines need testing (constraints don't compound). The model's own read is the one signal to avoid — naming a failure surfaces it. Residual: nothing in the corpus automates this; the per-release list is still hand-curated.
  • SourceDoes compounding require the instruction to name a behavior the model already has, or does any redundant instruction degrade output? Falsifiable by ablating one instruction at a time against a fixed eval. Partially answered 2026-08-04 by prompt design at scale (empirical) from the far side: redundancy is not required for degradation at all. Its 10-to-160 rules are distinct, non-overlapping and individually satisfiable, and all-rules compliance still floors by N=80 on all five models. So the answer to "does any redundant instruction degrade output" is bounded by a prior fact — instruction volume degrades the set regardless of redundancy, and the one-at-a-time ablation proposed here is exactly the method that cannot see it, since no single line is at fault. The redundancy half remains open and still needs the ablation.
• Knowledge-Centric Self-Improvement3 open
  • SourceDoes curation actually beat storage? The paper's conceptual separation from experience-reuse and memory systems (ExpeL, AgentKB, Agent Workflow Memory, Voyager) is argued in Related Work and never run as an arm, and no ablation removes the forums or distillation from its own protocol. A single arm — raw attempt table only, no forum, no distillation — would settle it. Sharpened rather than answered (2026-08-13): muscle memory compile not retrieve makes the compiled-over-retrieved argument its explicit thesis, names the same memory literature as the paradigm it displaces, and then also benchmarks only against a no-memory baseline — so the question now has two independent empirical papers arguing it and zero measuring it, and the unrun arm is the same one in both. Still #oq/source; the sharpening is that a source running it would settle two pages at once. Checked, not answered on this axis (2026-08-13): shopify sidekick continual learning loop was scouted as the counter-datum — a production system at 2,000 req/min that keeps accumulated experience rather than curating it, retraining daily "over the accumulated data" (new trajectories plus all previous ones) and framing that retention as what "limits drift and catastrophic forgetting across cycles." Read closely it does not answer this question, for two reasons worth recording so the next reader does not re-litigate them. (1) The two claims live in different spaces. Shopify accumulates in weight space, where retention is a training-dynamics constraint — dropping old trajectories from a fine-tuning corpus causes forgetting, an effect with no analogue for a text bundle read at inference, where a dropped claim costs nothing but the claim. This page's question is about context-space curation: whether selecting what a future agent reads beats handing it the raw log. Nothing about catastrophic forgetting bears on that. (2) On its own axis the source curates hard, in both directions. Entry to the training corpus is gated three times — the judge must score the conversation low (hard-negative selection), a frontier critic panel must produce a repair that makes the replay pass, and failures the critics cannot repair are escalated to human annotators rather than kept as-is — so what is retained indefinitely is an already-curated set, not raw traffic. And the artifact the deployed model actually reads is curated more aggressively than anything on this page: a system prompt compressed from ~6,000 tokens to ~1,500 learned gist tokens, plus prompt/tool/harness edits kept only when they raise the judge score. The one genuinely new datum is therefore not "storage beat curation" but that a production system runs both at once, on different objects — which is a reason to suspect the binary is mis-posed at the level this bullet poses it, and no reason yet to move it.
  • SourceRun both self-improvement artifact classes under one protocol — an evolved harness and a distilled knowledge bundle on the same tasks, same budget, same held-out and cross-model splits. What Makes a Self-Improvement Artifact Transfer? predicts the harness gains vanish off the matched pathology while the bundle gains persist across solvers; the prediction is falsifiable and no source has run it. Sharpened to three arms (2026-08-13): shopify sidekick continual learning loop adds a weight-space arm to the comparison and reports the two it runs sequentially rather than against each other — harness-space optimization until it plateaus, then parameter-space — with no measurement of either alone and no counterfactual for the ordering. So the missing experiment is now three-way (evolved harness / distilled bundle / fine-tuned weights, one task set, one budget) and one production source asserts the ordering the experiment would test.
  • SourceDoes the curated base keep adding value past 10 generations, or does it saturate once the easy tasks are retired? Every run stops at 10 generations with solved tasks removed from the pool, so the reported curve confounds knowledge accumulation with a shrinking task pool.
  • ResolvedWhat distinguishes a self-improvement artifact that transfers from one that does not? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? — an artifact transfers exactly as far as the regularity it encodes extends. The fitted-to-task vs fitted-to-model binary is the measured special case: harness patches encode solver pathologies (transfer within the pathology class — Qwen 27B→397B +11.0 nearly loss-free, Gemini +13.5 across families on the shared pathology — and fail outside it), distilled insights encode domain regularities (reuse holds the domain fixed, so all eight cells survive solver churn). Cross-release instruction depreciation is the same phenomenon on the time axis (the synthesis carries the evidence), transferability can be enforced at write time (this paper's schemas are the mechanism), and what transfers when the artifact doesn't is the procedure. The controlled two-artifact test remains open above.
• Latent vs. Deterministic Space2 open
  • SourceTan asserts the "wrong side" diagnosis covers most AI-engineering bugs. Does any incident/failure taxonomy (agent postmortems, eval failure analyses) actually classify failures by computation-locus, and what fraction lands in each side? Partially answered 2026-08-03 by reason less verify more (empirical): the first failure analysis in the corpus that classifies by locus and attaches a fraction — on the τ²-bench airline domain, 78% of observed failures are silent wrong-state failures traceable to policy living in a prompt document rather than in the tool, and moving four rules to the deterministic side recovers +12.4pp. Three limits keep it partial: it is one benchmark domain, it classifies along one axis (policy compliance) rather than taxonomizing failures generally, and the paper's own negative controls show the fraction is set by how the tool layer was built, so it is not a population estimate for agent bugs at large. Nothing yet measures the other direction of the diagnosis — code hard-coding judgment that belonged in the model. Advanced further 2026-08-03 by Layerwise Omission Attribution (where facts go missing, empirical), which supplies the taxonomy half almost completely: nine layers covering the whole pipeline, split explicitly into deterministic software (L0-L3) and model behavior (L4-L8), with a waterfall that assigns every lost fact to exactly one locus and a fixed order preventing double-counting. Its answer to the fraction half is 73.4% software — but that number comes from an allocation of deliberately injected faults, which the paper fences four separate times, so the taxonomy transfers and the fraction does not. Still open: any locus split measured on organic incidents, and still nothing on the reverse direction.
  • WaitThe seating example prices latent-space judgment at "a couple hundred dollars of tokens" for 800 seat assignments. As models absorb more deterministic capability (Harness Shrinkage as Models Improve), does the economically-optimal boundary move toward latent space, or does state-out-of-context remain invariant?
• Layerwise Omission Attribution3 open
  • SourceUnder organic production faults rather than Phase A's deliberate injection, does the L0-L3 software share stay above the L4-L8 model share at all? The 73.4% figure is a property of the designed cell allocation and the paper fences it four times; nothing in the corpus measures the layer split on unweighted production traffic. Distinct from Failures That Look Like Success's prevalence question, which asks what fraction of failures are silent rather than which layer produced them.
  • SourceDoes ω4 = 0.000 survive a workload that forces real context overflow? Engine/memory contributed zero attributed loss while q4 KV cache carried OR 2.25, because L4 is credited only when an event log shows the needle sat in an evicted range and corruption of a resident token scores at L5. Cheaply falsifiable: rerun the B1/B2 contrasts with payloads sized to evict the needle span and check whether the L4/L5 split moves. If it does not, the zero is an artifact of the attribution rule and the taxonomy needs a fifth deterministic class for corrupted-but-resident tokens.
  • SourceDo the L0-L3 checkpoint taps transfer to a hosted-API agent pipeline? The deterministic half of the method depends on reading raw bytes and token IDs at every boundary — T4 requires the final token-ID sequence, T5 requires KV-residency from engine logs — and no frontier inference API exposes either. If they do not transfer, the exactly-countable layers are auditable only on-prem, and the stack most agents actually run on is the one that cannot be instrumented.
• LLM-as-Compiler Knowledge Base5 open
  • SourceAt what scale does the no-vector-database approach break down? Karpathy's ~100 articles fit in context, but what about 1,000+? Partially answered (2026-08-13), on the design axis only: muscle memory compile not retrieve argues the compile side's per-call cost is fixed — one feature-extraction call plus one embedding lookup, "independent of how many patterns have been compiled" — against retrieval's 500–2,000 tokens of prompt inflation per call growing with the breadth of the store, and it relocates rather than removes the index (one 768-dim scope embedding per compiled artifact instead of one per fragment). It measures none of this. Its swarms cap at 1–6 artifacts per user (23 total), are never stress-tested larger, and it defers the exact comparison to future work in its own words; the bounded-routing claim sits in its Discussion and not in its Limitations. So the architectural argument now has an external statement and the scale number still has no source.
  • SourceWhat's the optimal granularity for concept articles — one concept per article, or clustered by theme? Partially answered (2026-08-03): Knowledge-Centric Self-Improvement answers a machine-read version of this and its answer is neither — granularity is carried by scope conditions attached to each claim (applies_when / does_not_apply_when) rather than by article size, with two levels of store (per-task and cross-task) whose inputs are kept disjoint. It also supplies a measured caution: transferring a fixed quantity of knowledge made recipient memory "noisy or detrimental," so their adapter bounds delivery at 0-3 items per field and returns empty lists when the prior is weakly relevant. Suggestive, not settling — bundles consumed by a solver agent are not articles read by a human.
  • SourceHow effective is the synthetic training data → fine-tuning pipeline in practice? Partially answered (2026-08-13), on the downstream half only: shopify sidekick continual learning loop runs the compile-into-weights step in production and reports a scaling curve for it — judge score 61.5 at 13k compiled trajectories rising to 73.5 at 61k, past the incumbent production system between 26k and 30k and level with a frontier reference at the top — so "does a domain fine-tune over your own accumulated material beat a general model on that domain" has an affirmative case-study answer with a dataset-size number attached. Three things keep it partial. The generation step is absent: their training data is repaired production trajectories, not synthetic text derived from documents, which is the half of Karpathy's proposal that is actually uncertain. The object compiled is behaviour, not knowledge, so nothing establishes that document-derived facts fine-tune as well as action trajectories do. And it is one unreplicated first-party account whose top point clears its frontier reference by 0.4 judge points with no intervals.
  • WaitDoes conformance to a knowledge format predict anything about the knowledge? OKF standardizes the container and explicitly assigns provenance, contradiction handling, and pruning to the producer, so a conformant bundle can be an uncurated dump — and its consumer half has no way to tell. The trigger event that settles it: the first bundle from a producer outside Google. Check whether it carries per-claim source attribution and any staleness signal beyond the single timestamp: field. If it does not, "portable format" and "worth retrieving from" are fully orthogonal, and the spec's minimalism is why.
  • SourceWould a pre-deployment quality gate on a compiled page catch errors a post-hoc structural lint cannot? Two sources now gate compiled artifacts before they run (a critic pass plus a held-out mini-eval; auto-generated acceptance tests from traces), and one of them measured the failure the gate misses — a fabricated API argument baked into a specialist's prompt, scoring 1/4 against the un-compiled baseline's 4/4. This vault gates nothing at compile time. The falsifiable version: run a content gate over newly compiled pages (does every quoted figure reconcile against its cited source region) and count how many defects it catches that lint.py structurally cannot see.
  • ResolvedHow to handle conflicting information across sources during compilation? Answered: When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time — a five-step protocol extracted from this vault's own practice and worked cases: (1) align constructs before declaring conflict (most contradictions dissolve into metric/population/time-axis/unit non-comparability — the Faros-vs-CMU worked example); (2) attach provenance and evidence tier, weigh by method and incentive, never average; (3) stage genuine conflicts explicitly on every affected page, bidirectionally — silent choice is the compile-time form of laundering; (4) convert staged conflicts into tracked open questions with named resolution conditions; (5) resolve at compile/lint time (Tan's librarian), so queries inherit the staged conflict with weights visible rather than re-adjudicating per query.
• Loop Engineering3 open
  • SourceOsmani's cost caveat is unquantified: at what token budget does a continuously-running loop stop paying for itself, and how do you instrument that? (Cf. Agent Loop Pattern's "who owns the budget when the model schedules its own loops.") Partially answered (magnitude only): a PostHog engineer running review/CI loops puts it at ~60% of personal token spend with no regret (Risk-Tiered Auto-Approval, case-study) — which establishes that the share can be a majority and still be judged worth it, but is a self-assessed share, not a break-even threshold or an instrument for finding one.
  • SourceIf /goal's stop-check is itself a model, what verifies the verifier? The maker/checker split pushes the trust problem up a level, not away. Partially answered (2026-08-04) — the regress is bounded rather than closed: Wu et al. (empirical) don't verify the verifier; they make its unreliability a measurable scalar (Youden's J = 1 − ρ₀ − ρ₁) and show what each level of it permits. Above roughly J ≥ 0.18 a loop can act on a calibrated estimate of the checker's noise and stay within 2.8pp of the true-parameter reference; at J = 0.03 the label-free calibration collapses (and collapses further with more data), and the correct move is to stop trusting any estimate and fall back to an estimation-free keep-best rule. What is still open is the part the question actually asks: measuring J at all needs a small labeled probe, so the trust problem is relocated to a human-labeled sample rather than dissolved. A second partial answer (2026-08-12), from practice rather than theory: a production security team's answer to the regress is to staff it — Trail of Bits (case-study) runs a security gate, then two judges on different models that must agree (one for plausibility, one PoC-focused and demanding reproducibility), then a human filter and a duplicate check, and separately builds a non-model instrument (aicov line-read coverage) for the one property a self-report cannot establish. That is the shape of a real answer — model diversity plus one exogenous measurement plus a human at the end — but the post publishes no false-positive rate for any stage, so it records what a team judged necessary, not what any of it caught.
  • WaitDoes loop-engineering converge on a single dominant shape (morning-triage → worktree → maker/checker → PR), or proliferate into many idiom-specific loops? The essay describes one shape "I keep using" but claims the primitives are general.
• MCP and Computer Use3 open
  • WaitThe MCP ecosystem's growth rate vs. computer use's quality curve: at what point does computer use become good enough that the marginal value of building an MCP server drops? Boris implies this is years off but doesn't quantify.
  • WaitIs computer use a sustainable interface or a transition technology? If most knowledge-work software adds MCP support in the next 24 months, computer use's role shrinks to legacy/desktop-only systems.
  • SourceMCP security model: as the playbook prescribes wiring MCP into Salesforce, Gmail, Calendar for solo founders, the attack surface scales with adoption. Partially answered by Zero Trust for AI Agents (tool poisoning, rug pulls, the first in-the-wild malicious MCP server) — see "MCP as a security surface" above. Open residual: how does a solo founder realistically run/host and self-sign every MCP server the framework recommends, given that the appeal of MCP was zero-integration-effort? Sharpened by ShareLock: the cheaper alternative to self-hosting — scan the tool descriptions with a guard model — is information-theoretically defeated by threshold fragmentation, so the lightweight mitigation doesn't hold and the burden falls back on run-your-own-server or downstream action-layer authorization. And Agentjacking shows run-your-own-server itself isn't sufficient: when the server is a legitimate observability platform relaying attacker-injected data (fake Sentry errors), self-hosting/vetting the server catches nothing — the untrusted input rides in on its data, so the residual burden falls squarely on the downstream data/action layer (provenance tracking + an out-of-band action gate), not on server hygiene.
  • ResolvedHow does Cowork's computer-use guardrail compare to Claude Code's auto-mode classifier? Different deployment context, possibly different risk profile. Answered: Classifier Gates vs OS Sandboxing: The Defense-in-Depth Story for Auto Mode and Cowork — same mechanism, inverted role. Cowork's guardrail is auto-mode-style classifier gating on the browser/computer-use surface (the Opus 5 card's browser row is measured on the Cowork harness: 31.5% bare → 3.70% → 0/129 scenarios with auto mode). The risk-profile difference is which layer can be load-bearing: Claude Code's blast surface is local and containable, so the sandbox can be primary and the classifier a convenience; Cowork drives the user's authenticated live SaaS sessions, where no sandbox equivalent exists and actions are less reversible — so the classifier carries the defense alone on the surface with the worst bare-model injection rate. Caveats: vendor-measured on a bounded suite, still a model-based gate (the D2 critique and the ADI forged-data failure shape apply), and the deterministic out-of-band action gate the research points to exists for neither surface yet.
• Open-Ended Discovery Harnesses4 open
  • SourceDoes the EvoX margin survive a matched budget? EvoX ran 100 iterations at ~$23.50/task against SwarmResearch's $50, and that is where the large wins are (2.635996 vs 2.1064 on circle packing; roughly doubled heuristics scores), while the budget-matched CORAL comparison is close enough that the authors call three of the wins polishable. Falsifiable directly: rerun EvoX to a $50 cap, on GPT-class models where its authors tuned it. Still open, and sharpened 2026-08-13 rather than answered. The obvious candidate answer — DarwinX, the one agent-search paper that argues the compute question head-on and labels its comparison "effort-controlled" — turns out to compare undefined vendor effort tiers (medium / high / xhigh are given no token, turn or wall-clock definition anywhere, its protocol appendix has no compute column, and it states no dollar cost), so it supplies no normalized budget and its own headline pair is not even tier-matched. The sharpened form of the question: this literature has three published attempts at the control and all three are categorical or partial, so the useful next experiment is not another comparison but a stated unit — dollars per task or rollouts per candidate, published by any of the competing methods, since without one no cross-paper margin in the class can be attributed to search design at all.
  • SourceThe architecture's structural half (fresh context, branch isolation) and its behavioral half (an LLM steering a population) are never separated. §3.5 reports the Shepherd defaulting to near-greedy and prescribing ideas against its own guardrail, and the Table 2 harness deliberately strips the behavioral half to a branch field and still wins — which suggests branch-preserving parent selection may be doing most of the work. Falsifiable: run the full 15-task benchmark with a random or heuristic parent-selector in the Shepherd's place.
  • SourceEvery method sits at 34–73% of human SOTA on all five contest-heuristics tasks, and the gap does not narrow with the better harness. Is that a search-procedure ceiling, or is ALE-Bench's Elo-like metric compressing a smaller objective gap into a large-looking rating gap? The paper raises the metric caveat and does not resolve it.
  • SourceDoes prescribing ideas collapse a population, or only prescribing ideas drawn from a source conditioned on the current best? SwarmResearch forbids it and measures its Shepherd going near-greedy anyway; Idea Search prescribes a stochastically-sampled, externally-seeded, score-tracked idea at every mutation and breaks a plateau with it. Falsifiable directly: give the Shepherd a decomposed idea bank and a sampler in place of its own judgment, keep everything else fixed, and see whether the near-greedy behaviour and the two lost tasks (EPLB, AHC026) change.
• Optimizer–Evaluator Decoupling3 open
  • SourceSingle-bit feedback bounds the leak per round, but not across rounds — a long enough accept/reject sequence is itself a channel into the sealed audit. Does SEAL's advantage survive a horizon much longer than the ten rounds tested (the one extended trace runs 21), or does acceptance rate and deployment truth decay as the agent accumulates bits?
  • SourceHow much independence is enough — different model family, different vendor, different modality of check (model judge vs. compiled test vs. production telemetry)? Partially answered (prescription, not measurement): PostHog (case-study) deploys the maximal-independence answer on all three axes at once — different instructions, different goals, and different models and providers per reviewer — on the stated rationale that agents are "unaware of their own blind spots." No ablation accompanies it, so it records what a practiced team judged necessary, not what is sufficient. Partially answered (measurement, one axis): HarnessBank ablates the modality axis — a deterministic evaluator plus a paired significance test, against the same loop crediting on mean improvement — and finds the difference shows up in archive quality and termination rather than in what ships (above). The model-family axis for the grader is untouched there, because its grader is not a model. Third axis, measured: disclosure. Guo et al.'s leaky-anchor arm holds the grader fixed and varies only whether its numeric scores are shown after a rejection — SEAL is at least as high in all six rows, strictly higher in five, with a 35.1 → 12.7 reversal on the worst cell. So independence is not one quantity: an equally independent grader is worth measurably less when the optimizer can read its numbers. Fourth axis, named but not measured (2026-08-03): Cursor varies the reviewer's evidence scope — full worker transcript, output only, or nothing but the codebase — alongside model, training and personality, and reports the design rule rather than the numbers: no single lens catches everything, decorrelated lenses stack. That converts the question from "how much independence" to "how uncorrelated are the failures," which is a set property and cannot be answered by grading one reviewer. Settling it needs per-lens catch rates on a shared bug set — the measurement neither production account has published. Fifth axis, measured, and it dominates the other four (2026-08-04): Zhou varies the grader's independence from the artifact rather than from the optimizer — commit an answer before conditioning on the candidate, or don't — and gets FPR 0.719 → 0.012 and discrimination 0.06 → 0.96 on identical text, with model family, scale and candidate-visibility all held fixed. That reorders the question's premises: on this task the axes this page has been sweeping buy less than the one it had not named, and the decorrelation hope takes a direct hit (three-family unanimous-accept ensemble still passes 55%; Proposition 2 rules out every monotone aggregation rule over a shared plausibility signal). Scope: an exact-matchable final answer is what makes the commitment checkable, so the result covers graders that can solve the task, not open-ended rubric grading. Sixth axis, measured in production but confounded (2026-08-04): Leni swaps the observe/compare stage of a live loop between a ~4B post-trained verifier and the frontier model that generated the artifact, and reports rescues 6 → 2 and correct rejection −4–5 pp. It is the first production measurement here, and the first where the grader sits inside the loop rather than after it — but it moves model family, model size, and post-training objective together, so it cannot say whether the effect is independence or specialisation, and the paper names the missing arm itself (an independent frontier model from a different provider). Single internal runs, vendor-evaluating-itself, two of four specialists. It sharpens the question's shape rather than its answer: "how much independence is enough" now has to be asked jointly with "how much of the observed benefit was never independence at all." The lineage axis, measured at last (2026-08-12), and it is the one this page had flagged as a hole rather than an axis: Greptile (case-study) varies only whether the grader shares the author's model family — harness, diff and ground truth held fixed — and finds each frontier model catches 6–12 fewer points of high-severity bugs in its own family's code, as a clean crossover with near-zero reviewer and dataset main effects. That answers the sub-question every prior entry deferred (different model family: yes, worth 6–12 points of recall) and reframes the Bun campaign's maximal-lineage configuration from a noted omission into a measurable cost. Three limits keep it from closing the bullet: it is a vendor's own labelled set with no released artifact and no judge validation, the effect is measured on code review rather than on optimizer-loop crediting, and it says nothing about vendor-versus-family granularity or about whether an open-weight third party sits inside or outside the cross-model band.
  • SourceThe seventh axis, unmeasured: a learned surrogate of an exogenous oracle. Jeff Dean (practitioner-opinion) prescribes replacing slow validators with neural approximations trained on the real simulator's output — a ~300,000× speedup at "nearly as accurate" for density functional theory — as the way to make automated experiment loops fast enough to matter (Recursive Self-Improvement). The surrogate is genuinely exogenous in provenance (trained from the oracle, not authored by the optimizer) but is an approximation with an error surface, and a loop running 10⁵ rounds against it optimizes that surface as readily as the objective. Where does a distilled oracle sit on this page's independence axes, and how many rounds does "nearly as accurate" survive? Nothing in the corpus measures it.
  • ResolvedDoes decoupling need to extend upstream to metric design? An optimizer that authors its own rubric has a subtler channel to game than one that merely reads scores. Answered (2026-08-03) by self authored verification unreliable (empirical): yes, and the question's framing was too generous. Guo et al. hand the optimizer both the policy and the test file and measure the divergence against a sealed deployment evaluation across six models and three seeds — 35 of 35 runs end with a self-score above 0.70 while 15 of 35 score below their game's random reference, and the per-model gap on Breakout reaches +0.92. The channel is not subtler gaming; the paper shows it is not gaming at all ("this does not require explicit cheating"), so an optimizer with a clean conscience produces the same divergence. Constraints that stay inside the self-authored instrument (monotone, discriminative) fall below no protection at all for four of six models, and the information limit α + β ≥ 1 - TV(P+, P-) says why: no endogenous-only gate can make both errors small once the regressing and non-regressing worlds look alike from inside. The sufficient fix is one sealed exogenous acceptance bit (SEAL), not honesty. Scope caveat: the instrument here is an executable test suite over programmatic Atari policies, so the result covers metrics the agent authors and runs; an LLM-judged quality rubric is the untested neighbouring case.
• Orchestration Sets Token Economics3 open
  • SourceDoes the effect survive against a competent third-party baseline rather than a vendor's own frozen loop? Every magnitude here is one harness versus one deliberately-superseded predecessor from the same company. A configuration-level cross-harness measurement of the token dimension — the paper names this as natural future work — would separate "good harness" from "bad baseline." Partially answered (2026-08-04): Databricks' internal coding bench compares three shipped third-party harnesses — Claude Code, Codex, Pi — with the model held constant, and reports the same success rate at "2x less cost" for the minimal harness plus a 3.13× per-task context gap on Opus 4.8. No arm is a superseded baseline, so the strawman risk does not apply to it. But it is secondary reporting with no task counts, no variance and no per-arm quality table, and it measures context rather than billed cost for the harness comparison — so the direction survives a third-party test while the configuration-level token measurement still does not exist in the corpus.
  • SourceDoes harness leverage hold outside the narrow band it was fitted on? r = 0.99 spans baseline capability means of just 0.710–0.789 over six models on one vendor's task set; the interesting question is whether the slope flattens, steepens, or inverts at frontier capability, where the harness is competing against a model that can increasingly do the orchestration itself.
  • SourceDoes the "orchestration beats model choice as a cost lever" ordering survive on long-horizon coding workloads? The task set mirrors an enterprise assistant (grounding, workflows, tools, content) and the paper concedes results may differ on SWE-bench-class work, where turn counts are far higher and the quadratic history term should favor the harness more, not less. Partially answered (2026-08-04): Databricks' bench is that workload — real engineering tasks against a multi-million-line codebase — and the ordering holds directionally, with harness choice moving cost ~2× at equal success while the model menu in the same write-up spans $1.28–$2.09 per task (≈1.6×). The two spreads are not measured under matched conditions: The Register never says which harness the per-task dollar figures were run under, and the per-task context figures come from a different pair of arms than the 2× cost claim. Corroboration, not replication.
• Output Length Calibration3 open
  • SourceDoes the end-of-prompt reminder work because of position (closest to generation) or repetition (stated twice)? The guide prescribes pairing both and does not separate the effects — testable by ablating the top-level instruction. Partially answered 2026-08-04 by prompt design at scale (empirical, five models) on the position half only: moving an identical, un-repeated instruction block between the system prompt and the user turn changes adherence by up to 8.7pp at N=160 — a larger effect than prompt format for four of five models — so position alone is a real lever, not an artifact of restatement. Two limits on how far that carries: the paper's placement arms are single-placement (never both slots at once), so it cannot separate position from repetition in the paired configuration this question asks about; and the direction is model-specific (user-turn placement helped two models, hurt two, and did nothing for a fifth), so "closer to generation is better" is not a rule the corpus can assert.
  • SourceDoes an explicit conciseness instruction cost quality on tasks whose answer genuinely needs length, or does the model still finish the work and only cut padding? Anthropic asserts the latter for the verification case but not for this one.
  • WaitIf the next model ships better-calibrated defaults, today's conciseness instructions become tomorrow's compounding instructions (Instruction Compounding) — does length calibration go stale the way verification instructions did, and is there a way to write it so it degrades gracefully?
• Parallel Agent Orchestration5 open
  • Waitp99 OpenAI runtime of 71 agent-hours/day is a frontier preview inside an unusually favorable environment. Does external concurrency actually trend toward it as frictions fall, or is heavy parallelism specific to model-adjacent work?
  • SourceSummed-overlap runtime can exceed 24h/day — it measures agent effort, not human attention. What is the human's actual oversight load per concurrent agent, and where does it saturate (AI Brain Fry)? Sharpened: HAS-Bench reframes the shape rather than measuring the load — in a controlled (LLM-simulated) benchmark the value of human input is configuration-dependent and non-monotonic (a right-timing / right-channel / right-authority sweet spot; more agency brings diminishing and sometimes negative returns), so per-agent oversight is likely a returns-curve with a peak, not a linear cost that hits a wall — but it is single-human, single-task, so it does not measure real concurrent-oversight load.
  • SourceConcurrency is measured over one week. Is 5+-agent management a stable practice or a burst around specific large tasks?
  • SourceDoes real coordination content displace task budget the way a synthetic template does? RCWT's block is one hand-written mix of role/protocol text, agent messages, shared propositions and tool schemas, and it topically overlaps several of the facts it scores — while the coordination traffic this page documents is dense tool dumps, verbose transcripts, design docs and contradictory agent claims. The falsifiable form: rerun the fixed-budget sweep with the coordination block drawn from real multi-agent traces (Bun's worktree shards, Cursor's swarm), varying content type independently of token count, and check whether the cliff still lands at the same residual reserve. If the reserve is content-dependent, "measure your task's residual budget" is not yet a portable rule.
  • WaitClaude Code's shipped fan-out defaults (200 spawns/session, 20 concurrent, fewer than 15 workflow agents, depth 3) are stated with no rationale and no measurement. Do they correspond to anything Anthropic measured — a runaway-loop incidence curve, a quality-vs-agent-count sweep — or are they round numbers chosen to bound a pathology? Until that is answered the product/research convergence on this page stays convergence. (Trigger: Anthropic publishing telemetry or a rationale for the caps, or a third party sweeping agent count on the same harness the way OrchBench did in simulation.)
• Prompt-Cache Economics4 open
  • SourceDoes the two-tier step survive at production prefix sizes, or is ρ ≈ 0.85 the real steady state everywhere above the threshold? §3 measures ρ = 1.0 at 4k–8k tokens while the 94k and 262k workloads both converge to ~85%, and the "multi-server replication" mechanism invoked to reconcile them is never actually characterized.
  • SourceIs the implicit caching of large tools= arrays a documented, stable provider behavior or an artifact of one routing configuration? It materially changes what explicit cache_control is worth on tool-heavy agents, and was found by accident rather than by design.
  • SourceAt what mutated-fraction of the cached prefix does query-aware compression cross from saving to costing? The two production workloads bracket the axis at roughly 15% (saves 31%) and ~50% (costs 40.1%), but no source measures the curve between them.
  • SourceDo caching and learned prompt compression compose, substitute, or collide? A gist-compressed prefix is byte-stable by construction (ideal for a cache) and small enough to fall through this page's tier threshold (where a byte-identical prefix still misses). The falsifiable version: one static prefix served three ways on one stack — full + cache_control, gist-compressed, and gist-compressed + cache_control — reporting billed cost or GPU-hours and quality, so the composition is measured rather than argued. The only source with gist tokens in production self-hosts its weights and therefore never faces the write premium at all.
• Reasoning–Acting Interleaving (ReAct)2 open
  • SourceThe lecture claims the interleaved trace is "extremely interpretable" and that this lets humans trust the model's responses. This wiki's later sources treat verbalized reasoning as an unreliable account of the computation (Chain-of-Thought Monitorability, Invisible Reasoning (Filler-Token Latent Computation)). Is a ReAct trace more faithful than a plain chain-of-thought because each step is anchored to an observed tool result — or does grounding buy verifiability of the actions while leaving the thoughts exactly as post-hoc as before?
  • NowEnumerating the valid action set fails when the action space is large, which is where every real agent now lives. Does anything recover the guarantee at scale, or is a typed tool schema plus a retry the whole of the current answer? Partially answered (2026-08-17) by Guarantees That Degrade at Deployment: Action-Space Soundness, Admissibility Without Effect, and a Vendor-Coupled Security Framework. The disjunct is settled: a typed schema is capability gating and is strictly weaker than what enumeration gave — it validates shape, and the cross-framework audit found LangChain/LangGraph, LlamaIndex and the Stripe Agent Toolkit all ship it with no deterministic fail-closed check over concrete argument values by default. The guarantee is recovered by relocating it: ScopeGate's stage 1 ("Unlisted tools → DENY. Blocks model-discovered tools and misspelled variants from reaching side effects") is this page's technique moved out of the prompt into the runtime, where list size costs nothing and an out-of-set action becomes unexecutable rather than unlikely — and stages 2–5 then decide the property a prompt list never could. The retry half is also improved on: NetInjectBench's gate substitutes a fixed safe fallback rather than terminating, which is why blocking costs almost nothing (0.00% unsafe tool-action rate at 99.17% useful-action rate). What is not answered is the "at scale" half. The bound changed from context capacity to policy coverage, and no source measures a gate over a large tool surface — ScopeGate governs a handful of tools, NetInjectBench two, and it inherits its policy from an existing change-management record rather than authoring it. Rashidi's Gap 4 finds that no paper in a 39-paper execution-security corpus measures policy-authoring error at all.
• Repository Exploration Subagent5 open
  • SourceDoes the gain survive better main models? The same-model-exploration result suggests the architectural benefit is somewhat model-independent, but the trained-explorer margin may erode as frontier models get cheaper and better at staying in their smart zone unaided. The bitter-lesson question is unresolved.
  • SourcePrune vs. don't-pollute. SWE-Pruner removes context after the fact; FastContext avoids accumulating it. Are these complementary (prune the solver and delegate exploration) or substitutes? Not tested together. Sharpened (2026-08-03): Tool-Output Pruning moves pruning to the agent-environment boundary, so the two now differ by one turn of exposure rather than by accumulation, and both target the same exploratory reads — which makes overlap, not additivity, the default expectation. Still measured on no benchmark together.
  • SourceHow small can the explorer go? The authors flag 1.7B / 0.6B as future work — if the recipe holds, the explorer becomes nearly free and the architecture dominates.
  • SourceGenerality beyond Mini-SWE-Agent. Only one (deliberately minimal) main-agent scaffold is tested; richer harnesses with their own memory/subagent orchestration may already capture part of the benefit or interact differently.
  • SourcePatch-derived reward leakage. Training the explorer's reward on the gold patch's file/line ranges risks overfitting to where fixes landed rather than where evidence lives; the F1-vs-recall behavior partly mitigates this, but the proxy is imperfect.
• Retrieval Inside the Reasoning Chain2 open
  • SourceSearch-o1 measures its own effect by counting hedging tokens in the reasoning chain and finds them down substantially. Does the marker count track correctness, or only the presence of retrieved text — i.e. would injecting irrelevant-but-confident documents suppress the markers just as well? The lecture reports the correlation and no control.
  • SourceThe reason-in-documents step is a second model call per retrieved document. Nobody in this corpus prices it: is compress-then-reason cheaper or more expensive than dumping documents into a long context, once the extra calls are billed? Both this lecture and the 2026 pruning papers report tokens or accuracy and never a bill.
• Shared Harness, Differentiated Surfaces2 open
  • WaitDoes one harness with UX-layer differentiation stay viable as audience breadth grows, or does the abstraction tax (hidden sub-agents, 32 model options, Ultra buried in advanced settings) eventually force a re-split?
  • SourceAnthropic ships two products split by output type while OpenAI ships one merged surface — is that a durable architectural disagreement, or is Anthropic's split a May-2026 artifact that later sources show closing?
• Stopping Under a Noisy Verifier3 open
  • SourceThe damage probabilities that drive every result here (β = 0.615–0.938) come mostly from a deliberately corrupted repairer premise. What are α and β on unperturbed production loops — code repair against a real test suite, tool-call repair against an executor — where nobody injected anything? MATH-500 and BFCL-single decline without the injection, so the direction is not purely constructed, but no field measurement of these two parameters exists. Partially answered (2026-08-04): Leni's production instrumentation supplies the first one — a shipping spreadsheet recalculation loop measures c = 0.20, r = 0.75, f = 0/357 (≲1% at 95%), i.e. α ≈ 0.15 and a damage term β ≈ 0, and the loop is net-positive (+1.5 pp) with no stopping rule at all. So the harmful-repair regime is not the default outside a corrupted-premise construction. Three reasons this doesn't retire the question: the oracle is a deterministic re-execution engine (this page's own limits note that deterministic verifiers break the repeated-query estimator, so it is the easy corner); b has no field value precisely because f = 0 left no false alarm to break anything; and the loop caps at two iterations, so round-index decline is untested. The asked-for cases — LLM-judged code repair, tool-call repair against an executor — remain unmeasured, and this is a single vendor measuring its own system.
  • SourceDiagnosing "my verifier's J is too low to steer on" currently needs labels: the paper deliberately uses a held-out labeled separation test rather than the binomial-mixture EM, to avoid diagnosing a broken estimator with its own output — and the EM is precisely what degenerates as J → 0. Is there a label-free J diagnostic that stays honest at low J, or is a small labeled probe irreducible?
  • SourceThe rule is one-step myopic and lands 17pp below post-hoc round-2 selection on the non-stationary trace. Does a round-dependent (α_t, β_t) model recover interior peaks, or is the peak only locatable in hindsight because the mechanism change that creates it is unobservable at the time? The paper names this as future work.
• Ticket-Driven Agent Orchestration5 open
  • SourceWhat's the right granularity for ticket size when the unit is "what one agent does in one workspace"? The post implies "much larger units of work" become viable, but how does that interact with the agent.max_turns limit (default 20)?
  • SourceHow do you prevent a ticket-extension cascade when agents file follow-up tickets liberally? Is the only governance check human triage at the Todo-state queue?
  • SourceDoes this pattern generalize to non-software work (research, ops, content)? The DAG dependency model and prompt-as-policy file should transfer; the per-issue workspace doesn't obviously.
  • SourceWhen an agent gets a ticket "completely wrong" (mentioned in the post), how is the lesson fed back into the system? Symphony's answer is "add guardrails and skills" — what's the institutional process for that?
  • SourceHow does ticket-driven orchestration interact with sprint planning / OKRs / roadmap work that operates on aggregates of tickets? Does the abstraction collapse when tickets are scoped that small?
• Tool-Output Pruning3 open
  • SourceDoes an in-backbone pruner survive a billed-cost audit? It defeats the prefix cache over every tool-response span by design, substitutes a modified history that invalidates the suffix each turn, and raised API calls on both backbones while raising per-trajectory input tokens on one. The paper reports tokens and wall time and never money; Prompt-Cache Economics shows a 3× token reduction costing +40.1%. Sharpened, not answered (2026-08-13): marginal value estimation deep research is the corpus's second measured pruner and contains no dollar figure at all — it defines "cost" as summed input + output tokens (§A.4) and labels its cost column # Tokens + Runtime (s), so the convention is now documented as a field-wide one rather than one paper's omission. What it does supply is the missing exchange-rate term: the input/output split per configuration, which shows early pruning removing output tokens (uncacheable, 5× input on Anthropic's card) proportionally faster than input, and late pruning removing input only. That makes the sign of the token→dollar conversion stage-dependent and turns the question into a computable one rather than an unmeasured one — the falsifiable form is now a single billed run of one early and one late configuration on the same pipeline, with cache-write and cache-read tokens itemised.
  • SourceWhy does the same pruning head raise MiMo-V2-Flash's SWE-Bench resolve rate by 3.8 points and lower Qwen3-Coder-Next's by 1.2, when it helps both on the read-only benchmarks? With n = 2 backbones and no proposed mechanism, "pruning helps patch generation" is not yet a claim — a third backbone would settle whether the split tracks model scale, attention architecture, or agent-training recipe. Reframed rather than answered (2026-08-12): sparse attention selectivity counterfactual (empirical) runs the same question on a different compression lever — block-sparse attention and KV-cache eviction rather than tool-output pruning — with a dense-calibrated counterfactual instead of a resolve rate, and finds sign heterogeneity is the expected behaviour of a compression lever rather than an anomaly needing explanation: two models × two tasks × three discard ratios produce two sign reversals running in opposite directions, and its three preregistered pooled tests are null (p = 0.995, 0.771, 0.541) precisely because opposite-signed cells cancel. The transferable part is the variable it identifies. Sign is selected by the operating point — how much is discarded — and flips within a single fixed model–task pair as the ratio moves, which is a candidate this question's three hypotheses (model scale, attention architecture, training recipe) do not contain: two backbones pruned at the same nominal keep rate are not necessarily at the same effective operating point. Not an answer — different lever, different outcome metric, no shared backbone, and every model there is 7B–8B — but it does mean a third backbone at one operating point would probably not settle it.
  • SourceIs any per-line label-match metric usable for selecting a pruner? F1 inverted against the judge on both examined cases, and the paper's fallback is an LLM judge that is itself unvalidated in this role. Nobody has checked whether the inversion also holds against a judge-free downstream metric (resolve rate on a matched harness), which would decide whether the judge is measuring usability or its own preferences.
• Tree Search over Agent Trajectories (LATS)2 open
  • SourceLATS scores a state with a prompted judge plus a sample-frequency term, and the same course showed frequency-based selection is blind to rare-correct solutions. Does the judge half carry the search on hard problems, or does the frequency half dominate and collapse the exploration the method exists to create? An ablation of the two terms would settle it.
  • SourceThe reversibility assumption confines tree search to simulators. Is there a version that scores a candidate action without executing it — a learned or prompted transition model — or does search over real-world actions reduce to "explore only in a sandbox, then replay the winning trajectory"?