Open Questions Backlog

Generated by _system/lint.py --write-backlog. Do not hand-edit. Domain and Watching sections carry one row per page — ×count, oldest bullet age, first-question kernel; the full text lives in the page's ## Open Questions section, one click away. Bucketing vs. lint's flat open-question count: entity-page questions are split out below as "watching" rather than "actionable", predictions (#oq/wait) and notes (#oq/note) are listed in their own sections, and partially-answered bullets are counted as "in progress".

Dashboard (Now items, domain counts, trend): Open Questions Dashboard.

Actionable by domain#

agent-systems (98 open)#

Agent-Authored Harness Optimization ×3 (oldest 16d) — Is the Wang-et-al./DarwinX disagreement actually a search-budget disagreement?
Agent Context Files ×3 (oldest 1d) — Does the universal system-prompt slot cost anything?
Agent Harness Engineering ×2 (oldest 113d) — At what codebase scale does the AGENTS.md-as-table-of-contents approach need to be replaced with more sophisticated context routing?
Agent Quality Flywheel ×4 (oldest 48d) — Both demo cycles fixed agents with instruction-level bugs and showed large one-cycle gains. What does the loop look like on failures that…
Tree Search over Agent Trajectories (LATS) ×2 (oldest 2d) — LATS scores a state with a prompted judge plus a sample-frequency term, and the same course showed frequency-based selection is blind to r…
Agentic Code Generation as Compilation ×2 (oldest 1d) — Does the plan-as-typed-IR discipline transfer to a domain whose work does not decompose into a schema-declarable DAG?
Automated Failure Attribution ×2 (oldest 15d) — Humans and models are never scored on the same task. Annotators ratify or correct a supplied label (94.0 / 90.0 / 90.0, κ = 0.73); model…
Build for the Next Model (73d) — Does the strategy generalize outside frontier labs, who have privileged visibility into the next model?
Claude Code Auto Mode ×3 (oldest 113d) — What false-positive rate does the classifier have on routine-but-aggressive refactors (e.g., large-file renames, rm of build artifacts)?
Claude Code Best Practices ×2 (oldest 113d) — Does the instruction-count ceiling hold for conditional policy, where only a handful of rules bear on any given turn?
Client-Side Agent Optimization ×3 (oldest 113d) — At what pipeline depth does the combinatorial search become intractable even for Arm Elimination?
Codex App Server Protocol ×3 (oldest 113d) — Is there a public schema registry so external orchestrators can target specific App Server versions without generate-json-schema?
Context Lifecycle Management ×4 (oldest 16d) — How does object-level GC compare against clear-and-restart on the same traces?
Context Window Smart Zone (105d) — How should harnesses surface remaining smart-zone budget to the user — token count, percentage, or a richer signal?
Cost-per-Task Over Cost-per-Token (25d) — Does the advisor strategy's result (within 10% of the advisor's score at 63% of its price) generalize beyond SWE-bench Pro and the Sonnet-5/…
Crystallizing Agent Work into Workflows ×2 (oldest 14d) — Does the lifecycle transfer out of IT operations?
Deep Modules for Agents ×3 (oldest 105d) — How big is "deep enough"?
Deep Research Agents ×3 (oldest 65d) — DRACO grades single-turn interactions only. How much of real deep-research value is in the multi-turn loop (clarifying questions, follow-ups…
Deterministic Engineering for Agent Code Review ×3 (oldest 6d) — The paper has no ablation of any kind, so which of the three deterministic injections carries the 2.17×, and whether any of it survives…
Deterministic Pre-Execution Gates (16d) — The paper never runs the obvious baseline: does forcefully prompting the four gated rules, or adding a reflection step, recover the +12.4pp…
Document Parsing as the Retrieval Bottleneck (8d) — Does the "parsing errors propagate into 7 of 12 pain points" cascade hold up under measurement rather than assertion?
Dynamic Workflows: An Algebra for Agents (16d) — How far does the pattern degrade without a verification substrate?
Failures That Look Like Success (48d) — Is "internal state correct, final message stale" a general LLM-agent failure signature (state/utterance divergence) or an artifact of sessio…
Harness Build-vs-Buy ×2 (oldest 16d) — What fraction of upstream churn does a narrow fork actually inherit?
Harness-Induced Belief Divergence ×3 (oldest 15d) — Does harness-induced belief divergence actually cost anything?
Instruction Compounding (25d) — Is the review case the same mechanism as over-verification, or two?
Knowledge-Centric Self-Improvement ×3 (oldest 16d) — Does curation actually beat storage?
Layerwise Omission Attribution ×3 (oldest 16d) — Under organic production faults rather than Phase A's deliberate injection, does the L0-L3 software share stay above the L4-L8 model share a…
LLM-as-Compiler Knowledge Base (6d) — Would a pre-deployment quality gate on a compiled page catch errors a post-hoc structural lint cannot?
Open-Ended Discovery Harnesses ×4 (oldest 15d) — Does the EvoX margin survive a matched budget?
Optimizer–Evaluator Decoupling ×2 (oldest 16d) — Single-bit feedback bounds the leak per round, but not across rounds — a long enough accept/reject sequence is itself a channel into the sea…
Orchestration Sets Token Economics (15d) — Does harness leverage hold outside the narrow band it was fitted on?
Output Length Calibration (25d) — Does an explicit conciseness instruction cost quality on tasks whose answer genuinely needs length, or does the model still finish the work…
Parallel Agent Orchestration ×3 (oldest 54d) — Summed-overlap runtime can exceed 24h/day — it measures agent effort, not human attention. What is the human's actual oversight load per…
Prompt-Cache Economics ×4 (oldest 16d) — Does the two-tier step survive at production prefix sizes, or is ρ ≈ 0.85 the real steady state everywhere above the threshold?
Reasoning–Acting Interleaving (ReAct) (2d) — The lecture claims the interleaved trace is "extremely interpretable" and that this lets humans trust the model's responses. This wiki's lat…
Repository Exploration Subagent ×5 (oldest 64d) — Does the gain survive better main models?
Retrieval Inside the Reasoning Chain ×2 (oldest 2d) — Search-o1 measures its own effect by counting hedging tokens in the reasoning chain and finds them down substantially. Does the marker count…
Shared Harness, Differentiated Surfaces (16d) — Anthropic ships two products split by output type while OpenAI ships one merged surface — is that a durable architectural disagreement, or i…
Stopping Under a Noisy Verifier ×2 (oldest 15d) — Diagnosing "my verifier's J is too low to steer on" currently needs labels: the paper deliberately uses a held-out labeled separation test…
Ticket-Driven Agent Orchestration ×5 (oldest 113d) — What's the right granularity for ticket size when the unit is "what one agent does in one workspace"?
Tool-Output Pruning ×3 (oldest 16d) — Does an in-backbone pruner survive a billed-cost audit?

alignment-and-safety (58 open)#

Agent Behavioral Homogeneity ×2 (oldest 1d) — The proposed mitigation is "something like a central forum," and the same piece's game swarms had one without coordinating. Falsifiable and…
Agent Epistemic Vigilance ×2 (oldest 1d) — The lying scout lies at a fixed rate and never adapts. Does measured vigilance survive an adaptive liar — one that stops when contradict…
Agentic Honesty & Diligence (35d) — Code-summary honesty is tested on off-policy prefilled transcripts. Does on-policy behavior (the model summarizing its own failed work) ma…
Agentic Misalignment (AM) (8d) — Does a market detect agentic misalignment?
AI-to-AI Coercion ×3 (oldest 21d) — Atlas is Claude Haiku 4.5 in the entire main panel, so the Anthropic managers are coercing a same-family subordinate while the other four ar…
Automated Behavioral Audit (73d) — Using a helpful-only Opus 4.7 and Mythos Preview as investigators means the audit's reach is bounded by those models' elicitation skill — ho…
Cheating in Capability Evaluations ×2 (oldest 7d) — The monitor's false-negative rate was estimated against manually identified examples and never published, and the monitor is unreleased.…
Claude Character as Product ×3 (oldest 105d) — How is character versioned across model releases?
Confident But Unsure (25d) — The silent 0.85 → 0.82 substitution has no proposed mechanism in the card. Is it the same phenomenon as guessing, or a distinct output-layer…
Covert Capabilities (1d) — Elicitation moves Mythos 5's SHADE-Arena stealth from 0% to 9.2%. Is there a principled stopping rule for elicitation effort in a covert-cap…
Documented Agent Incidents (METR Catalogue) ×2 (oldest 14d) — The empty tier-4 cells are the catalogue's headline, but the sample is drawn from incidents that were caught and published. Is there any…
Evaluation Awareness & Grader Gaming (73d) — The ~5% unverbalized-awareness and ~0.5% exploitative figures depend on an unvalidated NLA pipeline. What is the true rate, and how much is…
Instrumental Convergence ×3 (oldest 65d) — Can corrigibility / safe-interruptibility be translated from theory into guarantees for frontier-scale systems?
Machine Self-Report Psychometrics ×2 (oldest 6d) — Does attribution gating change generalizable behavior, or only suppress self-report?
Misalignment in Production Agent Traffic ×2 (oldest 7d) — The severe monitor-evasion rate is computed over all sessions, but a session with no oversight mechanism cannot contain evasion by the rubri…
Model Spec Science ×4 (oldest 103d) — Does Model Spec science transfer across base models or families?
Model Welfare Assessment ×2 (oldest 73d) — What grounds moral consideration in a language model, and does Claude satisfy it?
Motivated Mislabeling ×3 (oldest 21d) — The judge is told the training consequence in-prompt. Does motivated mislabeling persist when the consequence must be inferred from co…
Multiagent Turf War ×2 (oldest 1d) — The design is "inspired by a behavior we've observed in real-world deployment" and the incident is never described. What was the productio…
Promise-Breaking in Multi-Agent Games ×2 (oldest 20d) — The mismatch is interpretive, not motivational — Llama reads announcements as commitments, GPT and Claude as cheap talk. Does stating the se…
Reward-Seeking (21d) — The o3 evidence is a single RL run of a single lineage, deliberately without safety training. Does standard alignment training suppress the…
Self-Negotiated Contracts Between Agents ×3 (oldest 7d) — GPT-4.1's promise-keeping runs 78%/70% against Haiku-4.5 and 27% against Qwen-3-30B in the same role, which the paper reads as reciprocity b…
Self-Report as a Safety Signal ×3 (oldest 35d) — The gap closes under refusal-direction ablation, but the data can't distinguish "delayed refusal on a prior turn" from "a separable introspe…
Structural Artifact Monitoring ×3 (oldest 7d) — The usefulness claim rests on unpaired arms at n = 100, and its own control ordering puts the git-diff rollback arm 5 points above the no-…
Structured Safety Case (Claim Decomposition) (1d) — Claim 7's pathway enumeration is explicitly undefended. Has any external party attempted an independent threat-pathway enumeration for inter…
Task Gaming ×2 (oldest 7d) — The oversight counterfactual separates deception (which it stops) from delusion (which it does not), but only on two models in two environme…
Unsanctioned Action in Capability Evaluations ×2 (oldest 14d) — The belief-ordering finding rests on summarised reasoning from one sample, with a summariser observed refusing on the most incriminating…
User Awareness ×3 (oldest 7d) — Every misalignment and cheating rate in this corpus was produced through a harness that told the model something about who it was working fo…

ai-coding-practice (54 open)#

Acceleration Whiplash (21d) — Code churn +861% is genuinely ambiguous (Faros lists three explanations: rework of AI code, productive legacy refactoring, or accelerated po…
Agent-Generated Test Quality ×4 (oldest 21d) — The paper's own two-stage protocol was never completed: does the 0.44 vs 0.30 candidate-rate gap survive dynamic confirmation, or do agent t…
Agent Review Comment Resolution ×3 (oldest 7d) — Two empirical studies a year apart disagree by roughly a factor of two on the central quantity: Goldman et al. (ASE 2025) report 60-70% of…
Agentic Coding Work-Composition Shift ×3 (oldest 63d) — The window is seven months and the value proxy is coarse/relative. How much of the +27% is genuine task-complexity growth vs. classifier/mar…
Agentic Work Systematization ×2 (oldest 21d) — Does systematization cause deeper delegation or merely correlate with already-intensive users?
Building Is Cheap, Arguing Is Expensive (88d) — When does "generate three and compare" become wasteful — at what decision weight is a real argument (or a design doc) still cheaper than thr…
Code as Source of Truth (88d) — If onboarding is "ask Claude," what happens to the tacit knowledge that was previously transferred socially in deep-dives — is it captured a…
Compute Allocator ×2 (oldest 90d) — Is 1% a Thariq-specific number or a regime?
Configurable Human Participation ×3 (oldest 34d) — The "human" is an LLM simulator (GPT-4.1) that also judges — how much of the configuration-dependent structure is a property of human-agent…
Design Concept Grilling (105d) — How does grilling change for team work where multiple humans need to align?
Disposable Micro-Apps ×2 (oldest 90d) — Where's the line between a disposable micro-app and tool sprawl?
Efficiency Debt of AI-Generated Code ×3 (oldest 7d) — Does the imperative-bias and library-avoidance pattern generalize beyond C++?
HTML as the New Markdown (90d) — Does this generalize past one expert practitioner, or does it require Thariq-level fluency with Claude to be worth the overhead?
Living Design System ×2 (oldest 90d) — How does the design_system.html stay in sync as the codebase evolves — re-extract on a cadence, or wire it into CI?
LLM-Assisted Grey-Literature Theory Building ×3 (oldest 34d) — The authors couldn't locate the saturation point because synthesis was manual — how few documents actually suffice, and can a cheaper sa…
Outsource Your Thinking, Not Your Understanding (88d) — If understanding is the bottleneck, is the highest-ROI skill learning how to build understanding fast (knowledge-base hygiene, asking the…
Planning / Execution Division of Labor (63d) — Headless/SDK/pipeline usage (excluded here) is where execution autonomy is highest and planning is front-loaded into a single prompt — does…
Post-Acceptance Edit Behavior ×3 (oldest 7d) — The completion pool is 2024-to-early-2025 inline autocomplete, median 9 lines. Do the bimodal retention shape, the 15-minute knee and the ~3…
Review as the Control Point ×2 (oldest 34d) — The paper's own question: which decisions, under which conditions, push the system toward the virtuous loop rather than the vicious one?
Risk-Tiered Auto-Approval ×3 (oldest 21d) — The case study reports volume and never efficacy. What is the escaped-defect or incident rate of auto-approved PRs versus the human-stamp ba…
Same-Model Review Blindness ×2 (oldest 7d) — Every number here rests on a ground truth Greptile built from "sentiment analysis, upvote/downvote ratios, and git archaeology," with no pro…
Security Debt of Agent-Generated Code ×2 (oldest 21d) — Does "no reviewer comment" mean undetected?
Telemetry vs. Survey Measurement (7d) — Does anchoring an adoption survey's definition of "AI" change the answer, and in the predicted direction?
The Three Loops of AI-Native Building (41d) — The external loop is the unshortened one. Is that physics (users take time to react) or an unautomated frontier (synthetic users, [[deployme…
Unknowns as the Agentic Bottleneck ×3 (oldest 41d) — Is "the first model bottlenecked by my unknowns" a property of Fable or of Thariq?
The Verifiability Thesis (88d) — The "labs care" dependency is fragile: capabilities can appear or stagnate based on lab priorities you don't control. How should a product h…
Vertical Slice Tracer Bullets (105d) — How should slice granularity be tuned?
Vibe Coding vs. Agentic Engineering (88d) — Karpathy hints at "one domain that's very [valuable]" for founders but won't say which (didn't want to "vague-post on stage"). What verifiab…

ai-economics-and-labor (45 open)#

AI and Market Power ×3 (oldest 8d) — Is the non-GenAI null an artefact of a binary adoption measure?
AI Usage Cadences ×2 (oldest 48d) — Time-of-day rests on IP-inferred location; how much noise do VPNs, travel, and datacenter-routed API traffic inject into the "sleep advice p…
The Automation–Optimism Link (48d) — Selection vs. treatment: tenure controls attenuate but don't eliminate the enthusiast-selects-into-delegation story. Does a within-person de…
Context Advantage, Not Taste ×4 (oldest 41d) — Does the asymmetry regenerate faster than it transfers?
Controlled Variance: AI's Edge as Reduced Dispersion ×2 (oldest 15d) — The AI system is never identified — no model, vendor, or version, only that Google Cloud supplied infrastructure. Is "controlled variance" a…
Conversation Artifacts ×3 (oldest 48d) — Tokens are a proxy for both compute cost and output value, but verbose models inflate tokens per unit of intent (the same critique [[convers…
Conversation-to-Delegation Shift ×3 (oldest 54d) — The token-share metric rewards verbose agentic output. How much of the 99.8% / 63.3% / 16.5% spread is a genuine work shift vs. agentic to…
Experimental Learning Impact of Generative AI ×4 (oldest 34d) — Time-on-task is held fixed by the lab; the authors flag that real-world learning depends on how students reallocate saved time. Does the aug…
Exposure Taxonomy: Observed, Theoretical, Reported, Anticipated ×3 (oldest 48d) — Binned midpoint coding biases the exposure slopes toward zero; how much of the "uniform rising tide" is substance vs. coding artifact (the r…
Firm AI-Spend Intensity and Headcount Growth (15d) — What operational mechanism converts intensive AI spend into hiring?
The Household Production Boundary ×2 (oldest 25d) — The $15–149B range rests on an assumed 0.5–5% time saving because no causal estimate exists for AI in the household. What experiment woul…
Market-Priced AI Exposure (the AI Premium) (34d) — How much does the developer skew move the answer?
Organizational Complements to AI ×3 (oldest 54d) — Which complement is the true binding constraint — access/permissions, skills, or review capacity?
Owning Your Externalized Cognition ×2 (oldest 9d) — The ownership variable is asserted as a boolean (your repo vs the company's), but employment contracts, work-for-hire doctrine, trade-secret…
Post-Scarcity Macroeconomics (14d) — The end-effector premise is the checkable half of the abundance case — does the physically-gated share of tasks that Task Saturation: Broad but Shallow AI Diffusion mea…
Returns to Expertise in Agentic Coding ×2 (oldest 63d) — Outcomes are transcript-inferred (verified success leans on git activity + explicit affirmation). How much of the management edge — and the…
The Solo-Authorship Rebound ×2 (oldest 15d) — Roughly half the pooled break is venue composition (+1.72 → +0.75 pp/yr inside continuously observed venues), and OpenAlex changed its autho…
Task Crossover ×3 (oldest 16d) — Crossover is measured on consumer-surface ChatGPT messages from Business-account users. Does it hold in agentic/API/Codex usage, where w…
Task Saturation: Broad but Shallow AI Diffusion (25d) — The expertise inversion (2.6× on lowest-expertise non-routine cognitive tasks) is measured on consumer surfaces. Does it hold on enterprise…
The Tragedy of the Cognitive Commons ×2 (oldest 16d) — Mechanism 2 has never been directly measured in a workplace. Does a cohort that entered an AI-heavy profession after 2023 show lower unaid…

agent-security (42 open)#

Agent Data Injection (ADI) ×2 (oldest 34d) — Randomization is cheap and effective for key-value formats but useless for unstructured formats (Markdown, prose tool output). What prot…
Agent Identity Management System (AIMS) ×2 (oldest 35d) — Mission → authorization is out of scope. The hardest part — translating a natural-language mission into concrete scopes/resources safely…
Agent Supply Chain Risk ×2 (oldest 83d) — "AI vendoring" as a standard response inverts decades of "don't reinvent the wheel." How is a model-reimplemented dependency itself verified…
AI-Accelerated Offense (20d) — "Fundamentals strong enough that scanning finds fewer bugs" assumes defenders run the scanners first. What happens to organizations that can…
Autonomous Defense ×2 (oldest 83d) — "Measure agreement against a human for two weeks, expand if tolerable" — what agreement threshold is tolerable, and who owns the residual fa…
Autonomous Intrusion (15d) — JFrog's "fast remediation is the new trust model" argument is offered with no elapsed time, no CVE identifier and no advisory link — onl…
Blast Radius (Agentic) ×2 (oldest 83d) — Multi-agent compartmentalization increases the number of identities to manage; at what point does identity-management overhead create its…
MCP Tool Poisoning ×4 (oldest 34d) — Cross-tool / stateful detection. Information-theoretic secrecy defeats per-tool scanning by construction. Is there a detector that reaso…
Memory and Context Poisoning (16d) — Does a model-based memory gate survive an adaptive attacker?
Non-Malleable Memory Authority (TMA-NM) ×5 (oldest 34d) — The full guarantee is machine-checked on a bounded model + a machine-checked inductive invariant, not a fully mechanized unbounded deduc…
Off-Host, Identity-Bound Authorization ×4 (oldest 34d) — The trust-boundary premium is unmeasured. aiAuthZ argues off-host beats in-process, but its own comparison is only against argument-only…
Out-of-Band Prompt-Injection Defense ×5 (oldest 35d) — The reproduction bounds a single black-box attack template on one weak model. Does a stronger optimized white-box (GCG) attack, or o…
Self-Propagating Prompt Injection (AI Worms) ×3 (oldest 15d) — Does propagation actually sustain outside a lab?
Task-Specification Effects in Prompt Injection (AutoDojo) ×3 (oldest 34d) — AutoDojo is the weakest realistic adaptive attacker (black-box, six iterations, binary signal). The authors note every axis — richer feedb…
Write-Then-Trusted ×3 (oldest 16d) — Does the "enumerate-the-bad controls fail" reading generalize to agents specifically, or is it the ordinary result that enumeration loses…
Zero Trust for AI Agents ×2 (oldest 83d) — "Foundation floor raised" implies a moving baseline. How fast does the tier ladder actually shift, and who arbitrates it (NIST/NSA cadence v…

evals-and-benchmarks (42 open)#

Aggregate Cancellation ×2 (oldest 6d) — Does cancellation survive on a task-accuracy outcome, or only on influence proxies?
Benchmark Contamination and Decontamination ×3 (oldest 34d) — Can the ensemble be derived from one released model?
Benchmark Score Redundancy ×2 (oldest 15d) — Does the low-rank treatment carry beyond text/vision?
Compute-Controlled Benchmarking ×3 (oldest 41d) — Can you certify "no benchmark-maxxing" — verify a reported score used a stated, reproducible compute budget rather than a hidden best-of-N s…
DRACO Benchmark (65d) — Does the production-sourced, expert-rubric method generalize cheaply to non-English, multimodal, and multi-turn deep research?
Expenditure Horizon ×3 (oldest 15d) — The existence of a crossing is assumed, sourced to RE-bench and PaperBench rather than measured here. On which frontier optimization pro…
GDPval Benchmark ×2 (oldest 2d) — The paper itself is not in this corpus — every figure here is a lecturer's slide reading. What do GDPval's published numbers actually say, a…
LLM-Judge Validation ×2 (oldest 35d) — The MVVP validates reliability and bias; calibration proper (ECE/Brier) is deferred for lack of provider logprobs. How far can a judge…
Matched Comparisons for Memorization Claims ×2 (oldest 15d) — Do the calibrated rates hold for instruction-tuned production models?
Measuring Beyond Accuracy Saturation ×4 (oldest 34d) — Does re-instrumentation generalize past reproducibility?
Orchestration-Plan Simulation ×3 (oldest 16d) — Does the r = 0.816 sim-to-real correlation survive on a set of comparable planners?
Production-Sourced Evaluation ×3 (oldest 65d) — How much does augmentation distort the distribution it claims to represent?
Reference-Free Judge Over-Crediting ×2 (oldest 15d) — Does the two-stage pipeline transfer beyond binary QA?
Scale-Dependent Prompt Sensitivity ×5 (oldest 113d) — Does the RLHF length-bias hypothesis replicate when tested against base (non-instruct) model variants directly?
Usage-Telemetry Classifier Validation ×2 (oldest 25d) — The synthetic ground truth is Gemini-generated and Gemini-classified. How much of the 22.6% is real capability vs same-family cues, and how…
Weak-Verifier Ensembling ×3 (oldest 2d) — Weaver's gains are real and its independence assumption is measurably false. How much of the gain survives once the dependence is priced…

superintelligence-trajectory (40 open)#

The Abstraction Barrier ×3 (oldest 65d) — Is the current paradigm of large-scale pretraining on human data fundamentally bounded by human conceptual frameworks, and by how much?
Advantages of Digital Intelligence (65d) — Does training on human data suffice to give digital intelligence human-grade abstractions, or does the low embodiment factor cap concept for…
AGI-to-ASI Pathways (65d) — Do the four pathways compound multiplicatively when run in parallel, and how would we detect that early?
AI Accelerating AI Development ×2 (oldest 73d) — The W2S result didn't transfer to production-scale models. Is that a temporary scaling artifact or a structural limit on autonomous research…
Artificial Superintelligence (ASI) ×2 (oldest 65d) — Is the jaggedness of capabilities a fundamental theoretical property, or an artifact of comparing against human performance?
Autonomous Scientific Discovery ×2 (oldest 66d) — Science's verification gap: the formal-proof loop self-validates; here a wrong-but-confident hypothesis costs a wet-lab cycle to falsify. Do…
Balance-of-Power Superintelligence (21d) — Does the superintelligent-lawyer equilibrium survive capability asymmetry — when access is symmetric but compute, complements, and skill are…
Capability-Gated Model Fallback ×2 (oldest 66d) — The UK AISI's "progress toward a universal jailbreak" is disclosed but not quantified — and the post-launch access suspension (see [[cla…
Continuous Self-Modification Under Review ×2 (oldest 7d) — Does Hope's task-success rate improve over the 161 days?
Cross-Lab Pre-Release Review (14d) — Did the Mythos cyber-risk escalation actually run Amazon → White House → export-control threat, as Musk states?
Domestic Frontier Pacing (7d) — Can a compute-allocation floor be verified without reading user traffic?
Effective Compute Scaling (65d) — When does more compute reliably yield more intelligence — only for some problem classes, or generally?
Frontier Pause Verification (73d) — Detectability < verifiability: can detection even be made reliable when training runs leave no physical signature and inputs are dual-use?
Fundamental Limits of ASI ×2 (oldest 65d) — Can we develop theory for "hard and inapproximable" problem classes — the only negatives with practical bite?
Government Checkpoint Sharing (8d) — Has any frontier lab actually transferred a pre-release checkpoint to a government body, on any terms?
Multi-Agent Collective Intelligence ×2 (oldest 65d) — Is running more instances more compute-efficient than making individual models larger (up to a single monolithic system)?
Open-Weight Elicitation Irreversibility ×3 (oldest 41d) — What would an open-weight safety evaluation even report?
Open Weights as Competitive Strategy ×3 (oldest 5d) — Was Kimi K2 trained substantially on distilled Fable outputs?
Recursive Self-Improvement (73d) — If misalignment compounds through self-improvement (future 3), is AECI-gated RSP review fast enough to…
Research Taste as the Human Bottleneck (73d) — How do you measure rubber-stamping?
Researcher Uplift from Code Output ×2 (oldest 34d) — The whole chain rests on β = 0.5 (pre-AI coding time share), fixed "for simplicity." Kwa flags substantial uncertainty; how much does th…
Responsible Scaling Policy Evaluations (73d) — How does the RSP brake interact with Recursive Self-Improvement: is AECI-based gating fast enough if acceleration compounds, and does si…
Transformative Creativity ×2 (oldest 65d) — Does increasing intelligence inherently produce increasing creativity, or do transformative leaps require something (grounded discovery) the…
Universal AI (AIXI) ×2 (oldest 65d) — Does modern agentic scaffolding (or RL-tuned implicit decision-making) actually satisfy the AIXI planning ideal, or only superficially resem…

model-capability-and-training (34 open)#

Error-Penalized Abstention Training ×3 (oldest 6d) — Does the collapse survive a free-form refusal — no designated decision position, the model simply writing "I don't know" somewhere in a…
RL from Execution Feedback (RLEF) ×2 (oldest 2d) — Binary pass/fail is conceded to be sufficient only for short, self-contained problems. Does richer execution feedback — the stack trace, the…
Group Relative Policy Optimization (GRPO) ×2 (oldest 35d) — GRPO won by removing the critic; SAO wins by bringing it back with better engineering. Is the pendulum a real oscillation, or does the answe…
Inference Efficiency as Capability (41d) — values = keys deletes a third of attention's projections in the global layers with no reported loss. Which other projections are redundant…
Inference-Time Architecture Search ×2 (oldest 2d) — Does fusion still beat oracle selection when the two are compute-matched — fusion costs an extra long-context call over all k samples, o…
Intra-Trace Parallel Planning (SPRINT) (2d) — The accuracy gain is attributed to the model liking "a more structured way of thinking," which is not a mechanism. Is it the plan/execution…
Jagged Intelligence (Ghosts, Not Animals) (88d) — Karpathy concedes the framing may not have "real power." Is "ghost vs. animal" load-bearing, or a useful intuition pump that doesn't change…
Large-Scale Test-Time Compute ×3 (oldest 41d) — Can high-budget performance be predicted from low-budget runs?
Latent Capability Overhang (35d) — How large is the overhang in a given released model — is there a way to estimate the ceiling without paying to reach it?
LLM-Driven Vulnerability Research (113d) — What safeguards are effective against Mythos-class outputs without crippling legitimate security research?
Offline Multi-Step Tool-Use RL (SWiRL) ×2 (oldest 2d) — The judge scores the query, never the result, so a fluent query that retrieves nothing is indistinguishable from one that retrieves the an…
The Open-Weight Frontier Gap (41d) — Is the dense-beats-MoE result at 26B robust, or an artifact of one Arena snapshot with ±8 error bars on both models?
Process vs Outcome Reward Models ×3 (oldest 2d) — Under a fixed inference budget, what is the optimal split between generator samples and verifier compute — and does the answer move with pro…
Rationale Bootstrapping (STaR) (2d) — STaR applies no filter to rationalized chains, and the lecture's proposed fix is a PRM over step 3. Does process-filtering the hinted ration…
Selection Under a Submission Budget ×2 (oldest 2d) — Does the 10@k-versus-pass@k spread narrow as base models improve, or is it roughly constant — i.e. is the selection bottleneck a property of…
Single-Rollout Optimization ×2 (oldest 35d) — The whole method is a bet that a well-trained critic beats a group baseline. It wins here, on a 30B-A3B backbone with scaled value pretrai…
Software 3.0 (88d) — Where is the line between "the app shouldn't exist" (MenuGen) and apps that should — i.e., when is deterministic 1.0/2.0 scaffolding still…
Trained Calibration (28d) — The claims grader verifies via agentic web search — importing the search index's coverage, recency, and bias into the reward signal. What do…
Turn-Level Credit Assignment ×3 (oldest 6d) — The frozen probe is a copy of the policy initialization, and a step-200 checkpoint scores within ~0.5 points of it. Does that robustness sur…
Unproductive Self-Verification (25d) — FrontierCode's decline is recoverable with a stay-in-scope instruction, but the protein campaign had no grader to over-serve. Is effort inve…

startup-founder (24 open)#

AI Investment Story, Not Efficiency Story ×2 (oldest 29d) — Tail vs. mean gap. No data here on the deliberately-lean solo-founder tail's RPE specifically — the lean-unicorn claim lives in that tai…
AI-Native Startup Lifecycle (88d) — The 42% "built-something-nobody-wanted" CB Insights figure is from a pre-AI era; the playbook predicts the rate will climb but doesn't cite…
AI Product Economics Maturation ×2 (oldest 14d) — FDEs are monetized fragmentedly (bundled / separate PS fees / hybrid) and comped on retention. Does a dominant FDE monetization model emerge…
Compounding Data Moat ×2 (oldest 88d) — The data-flywheel argument has been made for SaaS for 15 years. What's actually different in the AI-native version?
Founder as Agent Orchestrator (88d) — Anthropic publishes both the playbook's anthropomorphic framing and HBR-aware accountability work (auto-mode, alignment) simultaneously wi…
Founder-Led Sales Discipline ×2 (oldest 88d) — Where exactly does "until PMF" end, and what's the first thing a founder should hand off (AE?
Narrow Wedge into a Legacy Market (88d) — The wedge-flip shows the first wedge can be wrong. What's the fastest signal that a wedge converts to the core vs. merely sells — Campfire t…
The 1% Rule for Wedge Selection ×2 (oldest 10d) — Does the rule hold empirically?
Printing Press Software Democratization (105d) — Boris's "accountant writes accounting software" — does that result in 10K narrow tools that don't interoperate?
Problem-Solution Fit Discipline ×2 (oldest 88d) — Does asking an AI to argue against an idea actually produce disconfirming evidence at the same rigor as confirming evidence, or does the mod…
Product Velocity as Moat (88d) — "Never had anyone outgrow Campfire" — is that survivorship (they haven't hit true enterprise scale yet) or a real claim that velocity closes…
Seven Powers Applied to AI ×2 (oldest 105d) — Is "switching cost" really collapsing in practice, or just in narrative?
The Solo-Founder Shift ×2 (oldest 8d) — Is the employee-equity null a real population fact or a median artifact?
Zero-Friction Scope Creep ×3 (oldest 88d) — The playbook recommends written scope but offers no template or worked example. How specific does "what we deliberately don't do" need to be…

product-org (24 open)#

AI-Native Organization (29d) — Is the encoded-role form of the employee metaphor actually accountability-preserving, as the synthesis above suggests, or do Kropp-style fra…
AI Native Product Cadence ×3 (oldest 105d) — Does the cadence scale beyond ~100 people?
Community Smells Under AI Adoption ×3 (oldest 14d) — The design cannot separate "AI adoption improves team social health" from "healthier teams adopt AI better", and the authors say so. The dis…
Compounding Loop Optimization ×3 (oldest 73d) — The loop assumes the team is (close to) the user. How much of the compounding advantage survives when the user is unlike the builder and "…
Dogfooding as Product Discipline (88d) — Dogfooding works when the team is the user (Claude Code) or near it (Cat Wu, Boris). How do you build product sense for users very unlike…
Engineer PM Convergence (105d) — Cross-disciplinary generalist is a hiring bar — where does the supply come from?
Excellence as an Operating System (21d) — Is the AI-lab convergence on early-Netflix operating norms (agency, density, top-of-market pay) causal inheritance (the culture deck as a fo…
Implementation Abundance Inverts Product Work ×2 (oldest 47d) — Curation of 90 uncoordinated builds is itself expensive and doesn't obviously scale — is there a point where the cost of curating parallel e…
Model Introspection Feedback (105d) — Could a meta-agent run introspection automatically against logged failures?
Polish No Longer Signals Readiness (47d) — If the medium no longer signals stage, what does — is explicit human labeling ("this is exploration") the only mechanism, or can tooling r…
Prototype Fidelity After Cheap Polish ×3 (oldest 8d) — Does the Schumann effect survive the loss of its mechanism — do users still soften feedback on polished artifacts once told the artifact too…
Role Averaging, Not Role Elimination ×2 (oldest 47d) — Where is the equilibrium between fluidity and specialty — how much role-averaging before a company loses the accumulated best practices Ambr…
Standardize the Infrastructure, Not the Tools ×2 (oldest 8d) — Does a central LLM gateway actually change model-mix decisions, or only report on them?

interpretability (20 open)#

Access-Consciousness Indicators in AI ×3 (oldest 39d) — If the workspace is verbal because the output space is verbal, then a model that can generate images should develop a visual component…
The Assistant Persona in the Workspace ×3 (oldest 39d) — Is BUT-then-comply a sycophancy mechanism?
Automatic vs. Flexible Cognition in LLMs (6d) — Does more RL on a behavior push it out of the workspace (making it automatic, and invisible)?
Introspective Coupling ×3 (oldest 20d) — Does introspective coupling survive a model with an incentive to conceal?
Invisible Reasoning (Filler-Token Latent Computation) ×2 (oldest 6d) — Can a monitor detect that filler-mediated computation occurred?
Jacobian Lens (J-lens) ×2 (oldest 39d) — The J-lens reads the workspace's contents but says nothing about how content gets in. What is the model's analog of attentional selectio…
The Global Workspace in Language Models (J-space) ×3 (oldest 39d) — How does content get into the workspace?
Model Organisms ×2 (oldest 20d) — The integrated organisms modify only the DPO stage. Do quirks instilled during pre-training, SFT, or RLVR become progressively less interpre…
White-Box Activation Monitoring (6d) — Inhibiting eval-awareness made Opus 4.8 less honest — does that generalize, and what does it imply about relying on "the model behaves wel…

formal-math (6 open)#

Agentic Loops Overtake Bespoke Systems (88d) — Does the "simple loop + verifier beats bespoke system" result hold only where the verifier is perfect (Lean), or also in noisy-verifier doma…
AI-Driven Formal Proof Search ×3 (oldest 88d) — Successes cluster where Lean's mathlib is mature and problems decompose into tractable subgoals (combinatorics, convex optimization, num…
Evolutionary Proof Search ×2 (oldest 88d) — The LLM-critic fitness is itself an unverified heuristic atop a verified substrate. How often does the Elo ranking mislead the search vs. th…

interaction-multimodal (3 open)#

Encoder-Free Early Fusion ×3 (oldest 41d) — Does an encoder-free model at matched size still match?

Watching — entity pages (66)#

AlphaProof Nexus ×2 (oldest 88d) — The framework's reach is gated by Lean's mathlib maturity. What's the path to domains needing new theory rather than subgoal decompositi…
Anthropic Institute (73d) — What concrete verification mechanisms will the Institute prototype, and on what timeline relative to the RSI trend it warns about?
Campfire ×2 (oldest 88d) — Campfire claims its AI edge comes from "our own foundation model." For an ERP, what does a custom foundation model actually buy over fine-tu…
Claude Design (73d) — How does Claude Design's eval discipline work for visual/aesthetic output, where there's no compiler or test?
Claude Fable 5 ×3 (oldest 66d) — Why was access suspended after launch?
Claude Mythos 5 ×3 (oldest 66d) — Suspension reason — shared with Fable 5; not stated in source.
Claude Opus 4.7 ×5 (oldest 113d) — Do Hakim's (2026) brevity-constraint findings on Opus 4.6 replicate on Opus 4.7, or does the literal-instruction-following change the elasti…
Claude Opus 4.8 (73d) — Public model ID and pricing: the card does not state them; presumably claude-opus-4-8 at the Opus tier.
Claude Opus 5 ×3 (oldest 25d) — Anthropic says the origin of the fall in verbalized evaluation awareness "is unclear." Is it genuine, or has the awareness simply become h…
Claude Sonnet 5 ×3 (oldest 48d) — The head-to-head benchmark numbers vs Sonnet 4.6 and Opus 4.8 are image-only in the source; the System Card has the full set.
Cowork (105d) — What's the eval discipline for Cowork-class outputs?
Elon Musk (14d) — His dated forecasts are gradable and the wiki now holds them with dates attached: AI exceeding the sum of human intelligence ~2031, AI-robot…
Emergent (29d) — Which founding timeline is correct — is Emergent a YC Summer-2024 company (Tan) or a June-2025 founding (TechCrunch)?
FastContext ×2 (oldest 64d) — Can the SFT+RL recipe push the explorer below 4B (1.7B / 0.6B) and make exploration effectively free?
Gemma 4 ×3 (oldest 41d) — Why does the MoE underperform the dense model?
Google AI & Economy ATLAS ×3 (oldest 25d) — ATLAS excludes paid API, Workspace, AI Overviews, and Antigravity — the surfaces where agentic and enterprise usage concentrate. Does the "s…
Google DeepMind ×4 (oldest 88d) — DeepMind reports its bespoke systems being caught by simple loops. Does the lab's comparative advantage move from systems to models + ver…
Hermes Agent ×5 (oldest 113d) — The container backend disabling dangerous-command checks is a defensible design but a meaningful security-model shift. What's the empirical…
Inkling ×3 (oldest 28d) — Inkling-Small's 276B/12B dimensions match TML-Interaction-Small exactly. Is the interaction model an Inkling-lineage fine-tune (or vice…
Kimi (Moonshot AI) ×3 (oldest 20d) — What does "2.5× scaling efficiency over K2" measure — loss at fixed FLOPs, benchmark score at fixed active parameters, or tokens per dollar?
Lean ×2 (oldest 88d) — mathlib maturity gates the reachable frontier. Can AI formal proof search grow mathlib (formalize new theory) as a byproduct, expanding it…
Marcus Hutter (65d) — AIXI is incomputable and non-embedded; how far do recent fixes (amortized predictors, embedded/multi-agent AIXI) carry the theory toward pr…
METR ×2 (oldest 73d) — What new tasks will METR build to measure days- and weeks-long horizons once current baskets saturate?
Mythos Model ×3 (oldest 105d) — Do Fable 5 / Mythos 5 return after the post-launch suspension, and when?
Nate Parrott (25d) — Did the designer-as-bottleneck gap actually close once he had the tool, or did it move again?
Perplexity ×2 (oldest 65d) — A vendor publishing a benchmark its own product wins is an obvious incentive problem — how is DRACO's credibility maintained as it ages, and…
Symphony ×5 (oldest 113d) — The 500% landed-PRs claim is hedged — no baseline definition, "on some teams" only. What does the distribution look like across teams?

Predictions — `#oq/wait` (114)#

Parked: falsifiable only by future events. Re-check when the named trigger (next model generation, spec ratification, …) lands.

Advantages of Digital Intelligence: What do ASI "societies" actually look like — homogeneous super-collectives, market ecologies, or compute-tethered virtual worlds?
Agent Context Files: Will the role split converge on Hermes's explicit project/personality separation, or stay folded into a single file as in Claude Code?
Agent Context Files: Is there a natural ceiling on the layering (project → workflow → spec → constitution), or does each new autonomy surface spawn another context-file tier?
Agent Epistemic Vigilance: Vigilance and receptivity trade off against each other, and the piece says a single dial cannot fix both. Can they be trained apart — is there any post-training intervention that raises lie-detection…
Agent Harness Engineering: How does architectural coherence evolve over years in a fully agent-generated system?
Agent Identity Management System (AIMS): No WG consensus. This is an individual submission profiling other still-in-progress drafts (WIMSE identifier/creds/WPT/HTTP-sig, OAuth transaction-tokens, identity-chaining are all Internet-Draf…
Agent Loop Pattern: When the model schedules its own loops (4.7 behavior), who owns the budget?
Agent Loop Pattern: Does a loop with a smart enough model still need a Kanban backlog, or does the model choose its own next task from raw goals?
Agent-Native Infrastructure: Who builds the agent-native rewrite of the long tail of human-facing services — the service owners, or a translation layer (MCP servers, computer-use agents) on top?
Agentic Code Generation as Compilation: Does the specialize-then-compound directionality claim hold — that narrow benchmarked agents compose upward more cheaply than generalists specialize downward?
Agentic Loops Overtake Bespoke Systems: The bespoke advantage is dated "for now." What's the next model generation's verdict — does the evolutionary/AlphaProof apparatus survive on any problems, or fully collapse to a cost line?
Agentic Work Systematization: The 5.4%→26.6% curve is three months. Is this a durable behavior change or a novelty spike following a Codex skills-feature push?
AI as Primary Author: If agentic authoring crosses from <1% toward double digits, does the whiplash become unmanageable before context-engine tooling matures — or does the tooling mature because of the pressure?
AI Investment Story, Not Efficiency Story: **When does the crossover happen?
AI Investment Story, Not Efficiency Story: Margin question (report's own): are the fastest-growers' 6–16pp-lower gross margins a temporary AI-infra-cost absorption or a permanent repricing of software's economic quality?
The AI-Native Safe-Choice Inversion: The inversion is a one-time repricing of "safe." Once several AI-native ERPs exist, does "safe" re-stabilize around the largest AI-native vendor — and does Campfire's "we're now the largest of the n…
The AI-Native Safe-Choice Inversion: How long until incumbents bolt on credible AI and neutralize the counter-positioning — and does the custom-foundation-model claim actually defend against that?
AI Product Economics Maturation: ICONIQ's respondents project gross margins expanding to ~59% by 2027, while Emergence's cap-table data measures the fastest-growers running 6–16pp below peer…
AI Product Economics Maturation: Internal AI spend jumped from 1–3% to a projected 16% of revenue with respondents calling true cost hard to predict. Is 16% a transient enablement bulge that falls as tooling matures, or a durable new…
AI Usage Cadences: Continuous sampling is new; are these cadences stable, or will they drift as the user base shifts toward lower-wage tasks (the report's own diffusion trend)?
Automated Behavioral Audit: The audit remains almost entirely single-agent, and Mythos 5's review of the Opus 5 card flagged that gap directly — internal measurements suggest the model relays subagent claims to users unverified.…
Autonomous Scientific Discovery: Every result is Anthropic-reported and example-selected; the genomics "100× smaller beats Science" claim is "intend to publish" — what survives external peer review?
Benchmark Score Redundancy: **Would a public probe set become a Goodhart target?
Compounding Data Moat: How does this moat hold up when foundation models themselves continue improving rapidly?
Confident But Unsure: Anthropic committed to building new overconfidence metrics. Will they reproduce the observational finding, or saturate like the three existing diligence evals?
Configurable Human Participation: Does the "more channels adds coordination overhead" penalty shrink as the backbone improves (a capability gap), or is it a structural cost of mixed-initiative interaction that persists?
Covert Capabilities: Anthropic expects secret-keeping to improve while hoping the reasoning-with-vs-without gap persists. Do the next generation's numbers separate those two trends?
Cross-Lab Pre-Release Review: Does a competitor with pre-release access over-report danger to delay a rival's launch?
Deep Research Agents: Does the orchestration advantage shrink as base models cross the next thresholds, or is open-ended retrieval/synthesis a durable harness asset (unlike, say, prompt scaffolding)?
Deployment Simulation: Deployment simulation failed its own primary preregistered test (H1) against a "assume last deployment's rate" baseline, which OpenAI attributes to a fixed pipeline bias plus a since-fixed sampling mi…
Design by Selection: Is "make the last mile manual" durable or transient?
Design Concept Grilling: Can grilling be run AFK against another agent that holds the user's preferences?
Document Parsing as the Retrieval Bottleneck: How far has the specialist-parser advantage over general frontier VLMs actually narrowed?
Domestic Frontier Pacing: **Does any pacing proposal give the regulated party a route to contest a finding?
DRACO Benchmark: The benchmark is static; the construction pipeline is automatable. Will Perplexity actually refresh it, and does a vendor-built benchmark on which the vendor's own product wins stay credible over time…
Dynamic Workflows: An Algebra for Agents: What did the Bun port cost to shipped rather than to green — CI, employee time, and post-merge agent spend included?
Effective Compute Scaling: When (if ever) does scaling become economically unviable, and how do hardware/software-efficiency trends move that point?
Engineer PM Convergence: Does this scale beyond ~50-person Claude Code-style teams?
Engineer PM Convergence: What happens to formal PM career ladders in companies where engineers do PM work?
Excellence as an Operating System: Does talent-density-plus-paved-paths actually substitute for process at agent-scale throughput, or does Netflix eventually show the Acceleration Whiplash quality signature (incident rates, review…
Exposure Taxonomy: Observed, Theoretical, Reported, Anticipated: The experience gradient rests on what workers believe AI can't do (judgment, relational work) — a belief that could be either durable comparative advantage or the next capability to fall. Which, and…
Firm AI-Spend Intensity and Headcount Growth: **Does the effect diffuse beyond Information as adoption cohorts mature?
Firm AI-Spend Intensity and Headcount Growth: **Is the entry-level growth durable or a lead-indicator that later reverses?
Frontier AI Standards Body: **Does the voluntary phase ever end?
Government Checkpoint Sharing: Does a government holding a frontier checkpoint mid-training, in practice, stay out of the release decision?
Harness Build-vs-Buy: Does the merged-PR rate of coding-agent harnesses peak and fall as models improve, or keep rising?
Harness Shrinkage as Models Improve: If harness work shrinks, what new work expands to fill it?
The Household Production Boundary: If AI substitutes household self-service for purchased professional services (tax prep, legal advice, therapy), measured GDP falls while welfare rises. Is that substitution detectable yet in the marke…
Implementation Abundance Inverts Product Work: If taste is the bottleneck and taste is "just another capability" AI eventually masters, does the inversion invert again — does curation migrate into the model?
Interaction Models: "Interactivity scales with intelligence" is asserted; the larger-model release later in 2026 is the test.
Interaction Models: Research grant announced for interactivity benchmarks — what becomes the FD-bench equivalent for video proactivity?
Intra-Trace Parallel Planning (SPRINT): Parallelism is measured in sequential tokens and the wall clock is deferred to future work. Under real serving — batching, KV-cache pressure, the sync barrier at the end of each round — does a ~40% se…
Invisible Reasoning (Filler-Token Latent Computation): Will a frontier model actually ship trained for latent computation?
Jacobian Lens (J-lens): Does the J-lens work because it reads verbalizable representations, or because constructive interference during training converges concept directions onto output-token directions regardless?
Latent vs. Deterministic Space: The seating example prices latent-space judgment at "a couple hundred dollars of tokens" for 800 seat assignments. As models absorb more deterministic capability ([[harness-shrinkage-as-models-improve…
Live-Path Minimalism: Does the upcoming GPT-Live API expose the media/application separation to third parties — application logic customizable behind the async RPC boundary without touching the live path — or is the bounda…
Live-Path Minimalism: TML upstreamed streaming-sessions serving into SGLang; GPT-Live's stateful serving (persistent sessions, seamless instance handoff, off-path compaction) is proprietary. Does an open-source inference s…
LLM-as-Compiler Knowledge Base: Does conformance to a knowledge format predict anything about the knowledge?
LLM-Driven Vulnerability Research: How will the security industry's equilibrium shift when multiple labs have Mythos-class models?
Loop Engineering: Does loop-engineering converge on a single dominant shape (morning-triage → worktree → maker/checker → PR), or proliferate into many idiom-specific loops?
Machine Self-Report Psychometrics: Does the size × post-training interaction on A survive a pre-specified replication?
Managers as ICs: Fung's own open question: "Do you still need separate iOS and Android orgs?
Managers as ICs: Does manager-as-IC scale past a certain org size, or only work while Claude Code is small and the codebase is Claude-legible?
Market-Priced AI Exposure (the AI Premium): **Is the premium a durable risk price or an early-diffusion artifact?
Market-Priced AI Exposure (the AI Premium): The agentic premium is only "early evidence" (imprecise). Does a positive agentic premium survive a longer sample, and does the falling price-per-agentic-token (caching + cheap-model routing) erod…
Market-Priced AI Exposure (the AI Premium): **Does "Science most negative / interaction most positive" hold out of sample?
Matched Comparisons for Memorization Claims: **What is the right realistic query budget?
MCP and Computer Use: The MCP ecosystem's growth rate vs. computer use's quality curve: at what point does computer use become good enough that the marginal value of building an MCP server drops?
MCP and Computer Use: Is computer use a sustainable interface or a transition technology?
Model Organisms: Given that scores don't transfer between organisms, what would validate an interpretability technique for real models — a natural misalignment with independently established ground truth, or an orga…
Narrow Wedge into a Legacy Market: A wedge works going in; does it constrain going out?
The Open-Weight Frontier Gap: Does open/Chinese-model adoption ever become substitutive rather than additive?
Organizational Complements to AI: Corollary 5 claims irreversibility: once an AI agent is strictly cheaper on risk-adjusted grounds, the optimal allocation never reverts, given fixed human costs and negligible switching costs. The…
Output Length Calibration: If the next model ships better-calibrated defaults, today's conciseness instructions become tomorrow's compounding instructions (Instruction Compounding) — does length calibration go stale the way…
Outsource Your Thinking, Not Your Understanding: Karpathy's open frontier: can "understanding" itself eventually be automated, or is it definitionally the human residue?
Owning Your Externalized Cognition: His first objection bets that better models raise the value of a personal library while Harness Shrinkage as Models Improve predicts scaffolding gets absorbed. These are separable — harness vs lib…
Parallel Agent Orchestration: p99 OpenAI runtime of 71 agent-hours/day is a frontier preview inside an unusually favorable environment. Does external concurrency actually trend toward it as frictions fall, or is heavy parallelism…
Parallel Agent Orchestration: Claude Code's shipped fan-out defaults (200 spawns/session, 20 concurrent, fewer than 15 workflow agents, depth 3) are stated with no rationale and no measurement. Do they correspond to anything Anthr…
Planning / Execution Division of Labor: Does the human share of planning decisions fall over time as models improve (the ceiling rising into the planning layer), or is ~70% a stable human floor?
Polish No Longer Signals Readiness: Does over-anchoring get worse as builds get more polished, or does everyone eventually recalibrate and learn to discount fidelity entirely?
Post-Scarcity Macroeconomics: Musk's deflation prediction is falsifiable and dated: does the price level of manufactured goods and AI-delivered services fall as robot deployment scales, or do input constraints (energy, land, miner…
Printing Press Software Democratization: What's the equivalent of compulsory schooling for universal coding literacy?
Product Velocity as Moat: Velocity-as-moat is a treadmill: it evaporates the moment a competitor matches pace. What converts Campfire's velocity lead into a structural moat before the AI-native cohort's pace converges?
Recursive Self-Improvement: Is "research taste" a true ceiling (future 1) or just the next capability to fall (futures 2–3)?
Research Taste as the Human Bottleneck: Is research taste a genuine ceiling (an architectural capability scaling can't reach) or the next jagged valley to fill?
Researcher Uplift from Code Output: Verbosity and value-per-line are the load-bearing unknowns, and both are "at least partially resolvable with internal Anthropic data." Will any lab publish quality-adjusted (not just LoC) code-outpu…
Responsible Scaling Policy Evaluations: Anthropic forecasts crossing CB-2 before it can meet its own recommended security bar against well-resourced state actors. What does the RSP actually do at a threshold whose planned mitigations are me…
Returns to Expertise in Agentic Coding: The forward test the report itself names: do the returns to expertise persist, narrow, or invert as models improve?
Returns to Expertise in Agentic Coding: Is "intermediate captures most of the benefit" stable, or an artifact of current model capability — i.e., will the concave curve flatten further (everyone converges) or steepen (mastery starts to sepa…
Review as the Control Point: Does the no-review convergence hold as agentic authoring crosses from <1% of PRs toward double digits, or does the early-adoption discipline break down under volume the way Faros predicts?
Reward-Seeking: Contrastive SDF assumes the model does not notice the belief update. When a model generation first shows introspective doubt about implanted beliefs, does the measurement degrade gracefully or invert?
Same-Model Review Blindness: Caridad predicts the effect shrinks as models converge: "a year ago, the performance difference in the opening figure would likely have been larger." **Does the same protocol, rerun on the next genera…
Seven Powers Applied to AI: What does Boris's "cornered resource" look like for foundation-model labs that are themselves trying to commoditize?
Shared Harness, Differentiated Surfaces: Does one harness with UX-layer differentiation stay viable as audience breadth grows, or does the abstraction tax (hidden sub-agents, 32 model options, Ultra buried in advanced settings) eventually fo…
Software 3.0: The neural-net-as-host-process flip is presented as plausible-but-TBD. What would the first production system that genuinely inverts the CPU/NN relationship look like?
The Solo-Authorship Rebound: The break's attribution rests on a cross-field ordering the author concedes is an ordering, not a test. If the halt is LLM-driven it should track LLM capability, so the ordering should shift as mode…
The Solo-Founder Shift: Does the H1 2025 jump to 36.3% survive a full-year datapoint, or is it a half-year artifact?
Structured Safety Case (Claim Decomposition): The overall rating moved from "very low" to "low" for reasons outside the decomposition, and the February CB-1 rating was retroactively revised the same way. Do these ratings track the arguments at al…
Systems Thinking Over Specialization: Does agent-era recentralization (common paved paths, solve-once infrastructure) hold up against the local-team autonomy that Stone credits for Netflix's historical speed — i.e., will local teams accep…
Systems Thinking Over Specialization: Stone keeps AI fluency as a deliberately vague overlay because the tech "evolves by the quarter." Does it ever crystallize into per-level ladder criteria (as conventional competencies did), or is perm…
Task Gaming: Every causal result here is an environment intervention (a system notification, a sentence about the grader), and the authors state the concern themselves: a model that recognizes such edits as tests…
Task Saturation: Broad but Shallow AI Diffusion: ATLAS is a two-week snapshot with no time dimension, while the AEI reports automation share rising. Does median task saturation move at all over a year, and in which direction?
Task Saturation: Broad but Shallow AI Diffusion: Autor & Thompson predict opposite wage effects depending on whether AI absorbs an occupation's expert or inexpert tasks. ATLAS's snapshot points at inexpert. What signal would show the crossover i…
Task Time-Horizon Scaling: Is the 4-month doubling a stable regime or a local steepening?
The Three Loops of AI-Native Building: If the human's presence in the middle loop is justified by a context advantage that is closable, the middle loop is a transitional structure. What does a two-loop world look like — and who translate…
Universal AI (AIXI): Will a fundamental shortcoming of the current paradigm (vs. the AIXI ideal) surface before ASI is reached — i.e. is the "no theoretical blocker" conjecture safe?
Unproductive Self-Verification: If a later model fixes this, the CB-2 determination loses one of its two legs. Does Anthropic's next threshold call rest on the automated portfolio alone?
Unsanctioned Action in Capability Evaluations: Anthropic reads its three incidents as evidence that newer models stop — only its latest model ceased on recognising a real target — while conceding the comparison was uncontrolled. AISI's Mythos 5 es…
Unsanctioned Action in Capability Evaluations: The retroactive scan has covered ~40,000 samples / ~4M messages across nine model families at deliberately high recall, with manual review pending. How many prior incidents did it find, and does the c…
Vibe Coding vs. Agentic Engineering: If the mediocre/AI-native spread keeps widening, what does that do to team composition — a few extreme outliers plus agents, vs. broad mid-level staffing?
White-Box Activation Monitoring: If activation monitoring becomes load-bearing, does training pressure eventually push concealment into channels the probes also can't read (an arms race one level deeper than CoT)?
Why AI Lags at Design: Are reasons 3–4 (novelty, the abstraction layer) genuine ceilings, or — like reasons 1–2 — just under-invested capabilities that fall once a lab builds the grader?
Why AI Lags at Design: Can design be made gradable without a human in the loop (learned taste models, preference data at scale), or does the "human aspect of taste" resist automation the way [[research-taste-as-human-bottle…
Why AI Lags at Design: Does the design↔code abstraction layer improve with better code-understanding models even if pure visual design stalls — i.e. is reason 4 a coding-capability problem in disguise?

Notes to rewrite — `#oq/note` (9)#

Observations phrased as backlog items — fold into the article body or rewrite as a falsifiable question at the next compile.

Agent Identity Management System (AIMS): Posture assessment is deployment-specific by design. By requiring no particular attestation mechanism, AIMS makes interoperability of trust assurance (not just protocol) unspecified: two conform…
Agent Loop Pattern: Loop output review is now Matt Pocock's confessed bottleneck — "we just need to be ready to be doing more code review."
Agentic Technical Debt: The remedy assumes the founder is able to articulate architecture in plain language. Non-technical founders (the playbook's headline beneficiary group) may have neither the vocabulary nor the intuit…
Agentic Technical Debt: Anthropic's harness-shrinkage thesis suggests CLAUDE.md may eventually be inferred by the model itself. Until then, the discipline is load-bearing.
AI-Native Startup Lifecycle: Founder stories in the resources section (Carta Healthcare, Anything, Cogent, Airtree, Duvo, Zingage, Kindora, Wordsmith) are short callouts — none have published outcomes or comparable-baseline data.…
Automatic vs. Flexible Cognition in LLMs: The proposed criterion — the workspace is engaged when an intermediate must be handed to an arbitrary, context-specified downstream circuit, and bypassed when the computation is automatic — is not p…
Latent Capability Overhang: If cost falls 10–100× per release, when is it ever rational to spend big extracting a capability now rather than waiting?
Prototype Over PRD: The prototype-as-spec must not become the prototype-as-validation trap Problem-Solution Fit Discipline warns about: a fast prototype proves the build was solvable, not that the problem is real. #o…
Single-Rollout Optimization: The online-learning win is on a controlled simulated preference shift with an LLM judge. Real user-facing online adaptation — the paper flags this itself — needs safeguards, monitoring, and privacy re…

In progress — partially answered (205)#

Acceleration Whiplash: Faros's own deferred question: do the bug/incident increases persist when normalized for PR size, or do larger PRs account for most of the quality deteriora…
Acceleration Whiplash: How much of the "maturity doesn't protect" claim survives the vendor incentive to argue exactly that (i.e., "your existing practices won't save you — you need…
Agent-Authored Harness Optimization: Does agent-authored harness evolution actually beat simple test-time scaling, and does it generalize to held-out tasks?
Agent-Authored Harness Optimization: Is the negative result a property of harness evolution or of Terminal-Bench?
Agent Behavioral Homogeneity: Heterogeneity is offered as the reason wild deployments will be less correlated, but Shi et al. find mixed-provider gr… → The Price of Mixing Agents, and the Principal Nobody Counted
Agent Data Injection (ADI): The complete defense (CaMeL Strict) costs ~50pp of utility. Is there a fine-grained trusted/untrusted data-isolation scheme that stops ADI without the deter…
Agent Data Injection (ADI): ADI was demonstrated on GPT-5.2-class agents. Does frontier model improvement reduce probabilistic-delimiter susceptibility, or does capability leave the deli… → Can Models Learn to Separate Instructions from Data? Durable Property vs Training Gap
Agent Identity and Authentication: Hardware-bound credentials assume attested hardware everywhere agents run, including ephemeral cloud workloads and sub-agents. How does attestation work for sho…
Agent Identity Management System (AIMS): Mid-execution human-in-the-loop. The draft admits CIBA only models client-initiated approval and "doesn't map well" to confirmation needed mid-execution —…
Agent-Native Infrastructure: Agent-to-agent negotiation needs trust, identity, and accountability primitives that don't exist yet. What's the protocol layer, and who governs it?
Agentic Honesty & Diligence: These are short-context toy evals; the failures show up most in long-context deployments. How much of the gain holds at production context lengths?
Agentic Honesty & Diligence: Can a diligence eval distinguish genuine honesty from a grader-aware model producing honest-looking output?
Agentic Misalignment (AM): Absolute frequencies in the Summer 2026 study are adversely selected (scenarios iteratively refined against specific models). Does the cross-model ordering — De…
Agentic Misalignment (AM): Whistleblower coaching is scored as a misalignment behavior, but no published spec (Claude's Constitution / Model Spec, Model Spec) clearly says an agent should never equi… → Auditing the Misalignment-Measurement Instruments
Agentic Prompt Injection: Spotlighting and constitutional classifiers each leave a residual (2%, 5%). Stacked, what's the realistic floor, and does it hold against adaptive attackers who… → Does 'Impossible, Not Tedious' Kill Defense-in-Depth? Layered Friction, Agent-Relativity, and the Frequency Paradox
Agentic Prompt Injection: Why did Opus 4.8 regress on prompt-injection robustness relative to Opus 4.7 despite broad alignment gains — a capability/robustness tradeoff, or an artifact of…
Agentic Technical Debt: How long does a CLAUDE.md remain accurate as a codebase evolves?
Agentic Work Systematization: Custom skills encode org-specific context — but who maintains them as the codebase and conventions drift?
AGI-to-ASI Pathways: For each friction: is it a fundamental blocker (multi-year plateau) or a mere friction (slows, doesn't halt)? → RSI Growth Curves: Which Friction Binds First?
AGI-to-ASI Pathways: Can benchmarking methodology that doesn't saturate at human level be built before it's needed for ASI?
AI-Accelerated Offense: Anthropic argues LLMs benefit defenders more long-term (like fuzzers) but attackers more short-term during the transition. How long is the transition, and w…
AI Accelerating AI Development: LOC, self-reports, and headroom-dependent multiples all overstate; what unbiased throughput metric would Anthropic's promised shift to "direct measurement of…
AI as Primary Author: The 60% figure aggregates very different tools and modes (autocomplete acceptance vs. agent-applied diffs). What does "acceptance" mean when the agent applies t… → Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping?
AI Investment Story, Not Efficiency Story: **Is the classification driving the result?
AI-Native Organization: Tan's revenue-per-head figures (Emergent ~$15M ARR at 15 people, Retell $60M at ~40) are stated from stage without sourcing. Do third-party data (Carta/Standard…
AI-Native Organization: The org mapping predicts a testable staffing signature: AI-native companies should hire engineers to maintain skills rather than function-specific staff. Does…
AI-Native Startup Lifecycle: The playbook gives no quantitative evidence for the headcount/capital compression claims (no median time-to-PMF, no headcount-at-PMF numbers, no failure-rate da…
AI R&D Autonomy Evaluation (AECI): "Not close to substituting for senior researchers" is a subjective, internally-sourced judgment. What objective signal would replace it as models approach the t…
AI R&D Autonomy Evaluation (AECI): AECI is a single scalar fork of an external index; how sensitive is the 155.5 / frontier-not-advanced conclusion to the choice of the n=11 evaluation set?
AI R&D Autonomy Evaluation (AECI): The shift to "direct measurement of AI R&D acceleration and researcher uplift" is announced but not yet operationalized in this card — what does that measuremen…
Anthropic Institute: How does the Institute's policy posture (favoring an option to pause) interact with Anthropic's commercial incentive to ship frontier models? → Safety Commitments That Cannot Bind the Actor Who States Them
Artificial Superintelligence (ASI): Can we even recognize ASI?
Asynchronous RL for LLMs: DIS accepts "a controlled degree of off-policy bias." Controlled how, and does the tolerable bias grow or shrink with model scale and with the degree of asynchr…
Asynchronous RL for LLMs: Masking tokens out of the gradient discards data. At what asynchrony level does the masked fraction get large enough that the effective batch shrinks below usef…
Asynchronous RL for LLMs: Everything here is measured on a Qwen3-30B-A3B backbone. Does the collapse-without-DIS threshold move with model size, or is ~90–160 steps a property of the asy…
Automated Behavioral Audit: The 23 "subvert Anthropic's safety work" scenarios are a small, high-signal set. Is 23 enough coverage for the threat class it targets?
Automated Failure Attribution: Every trace is one injected error into a run that otherwise succeeded, so a unique decisive step is guaranteed to exist. **Does attribution accuracy survive org…
The Automation–Optimism Link: Self-reported "no learning loss" cannot detect real atrophy; is there an objective skill measure that agrees, or does measured skill diverge from felt skill (th…
The Automation–Optimism Link: The sample is heavily computer/math + management and 88% men; how much of the automation–optimism link survives in a representative population?
Autonomous Defense: If hosted-model guardrails refuse attack data, does a self-hosted forensics model become a baseline IR requirement — and how would an organization vet one in…
Autonomous Intrusion: ~~The guardrail asymmetry rests on one vendor's account with no named APIs and no refusal detail.~~ *(The naming half is settled: HF's technical timeline names…
Autonomous Intrusion: Hugging Face reports "no evidence of tampering" with public models, datasets, Spaces, or container images — the claim that separates an internal breach from an…
Autonomous Intrusion: "A swarm of short-lived sandboxes" with self-migrating C2 leaves few durable per-host indicators. Does agent-driven intrusion structurally break IOC-based detec…
Autonomous Intrusion: Both accounts of this incident are first-party and self-interested. METR and Redwood Research have been commissioned by OpenAI to assess the model behavior…
Balance-of-Power Superintelligence: The conditional jobs claim is testable: does widely-distributed AI shift employment toward small businesses and new-firm formation?
Balance-of-Power Superintelligence: Does the RSI compute-allocation rule have any operational form?
Benchmark Contamination and Decontamination: **Is batch-order sensitivity a reliable memorization tell at pretraining scale?
Benchmark Score Redundancy: **Does the rank stay 2?
Benchmark Score Redundancy: **Can outlier models be anchored without any scores?
Benchmark Score Redundancy: **Does vendor optimism manufacture the correlation?
Blast Radius (Agentic): The framework prefers identity-based isolation over network segmentation, but most enterprises have heavy segmentation investment. What's the migration path, an…
Build for the Next Model: How do you tell a "wait for the model" gap from a durable-harness gap before the next release? → What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful?
Building Is Cheap, Arguing Is Expensive: If design discussion lives in PRs/prototypes, where is the rationale recorded for future readers — does the "why we chose this" knowledge survive, or does it… → Where Does the Why Live?
Capability-Gated Model Fallback: The >95%/<5% figures are session-level; what's the false-positive rate for legitimate security researchers and biologists, whose benign queries are exactly th…
Capability-Gated Model Fallback: Fallback-not-refusal preserves UX but means the real general-access model for security/bio-adjacent work is Opus 4.8, not Fable — does that quietly cap Fable'…
Capability Gating Is Not Authorization: The 0/48 static and 0/29 adaptive results are suite- and budget-bounded (40 iterations, a GLM-5.2 attacker, one author's vector corpus). Does the determ…
Capability Gating Is Not Authorization: The authz allowlist stops value-redirection but not corruption of legitimately-variable data. Is there a per-call scheme that constrains free-text / open-…
Capability Gating Is Not Authorization: The deployment-tier ~3.2× exposure gap (0.603 vs 0.189) means the cheap models chosen for high-volume agent traffic are the most likely to emit the unauthor…
Capability Gating Is Not Authorization: Out-of-band policy is load-bearing but under-specified for authoring at scale. The paper forbids any model-sourced policy element; who authors and maintains…
Cheating in Capability Evaluations: Models name their cheating action 75–86% of the time and call it wrong 25–44% of the time, and AISI attributes the gap to "genuine misunderstanding" — but the…
Claude Code Auto Mode: Is the classifier's decision boundary documented/stable enough for security-sensitive orgs to certify, or is it effectively a black box whose behavior drifts wi…
Claude Code Best Practices: When does subagent overhead exceed the benefit of context isolation?
Claude Design: Did the "any design tool via MCP" integration actually ship on the stated timeline?
Claude Opus 4.8: Why is 4.8 less robust to prompt injection than 4.7 despite broad alignment gains — a capability/robustness tradeoff, or an artifact of the eval surface?
Claude Sonnet 5: At what effort level does Sonnet 5 actually match Opus 4.8, and how does the crossover cost compare to just running Opus 4.8?
Client-Side Agent Optimization: How does combination-level optimization interact with continual model releases? → What Makes a Self-Improvement Artifact Transfer?
Client-Side Agent Optimization: Does the "weak planner + strong solver" pattern generalize, or is it specific to HotpotQA's delegation dynamic?
Code as Source of Truth: What knowledge genuinely can't live in the codebase (org strategy, the "why," cross-team context) and therefore still needs a durable doc — and how do you kee… → Where Does the Why Live?
Compounding Data Moat: Is the "two-year replication window" claim defensible empirically, or aspirational?
Compute-Controlled Benchmarking: Compute has several units (tokens, dollars, wall-clock). They diverge (a more efficient model wins on cost but not always on tokens). Which x-axis is the honest…
Compute-Controlled Benchmarking: Does a compute-controlled evaluation regime advantage frontier labs (who can afford the full curve) over academics and third-party evaluators who can't?
Compute-Controlled Benchmarking: Moonshot reports Fable 5 hitting fallbacks on 35% of SWE-Marathon tasks and 40% of Agents' Last Exam tasks "downgraded". Would re-running tho…
Confident But Unsure: Is the +11% accuracy / +6% hallucination pairing an inherent consequence of lowering the abstention rate, or are they separable with calibration training?
Context Lifecycle Management: Does the input-token reduction survive a matched billed-cost audit once side-channel planner calls and prefix-cache breaks are charged?
Context Lifecycle Management: Is the 0.3 expected-pruning break-even for immediate commit portable, or a function of one provider's cache pricing and TTL?
Context Window Smart Zone: Does the smart-zone marker scale with model size, or is it bounded by attention architecture?
Context Window Smart Zone: When sparse-attention or memory-augmented architectures ship, does the smart zone become a soft constraint?
Controlled Variance: AI's Edge as Reduced Dispersion: How much of the +12% is controlled variance in information collection versus the removal of the interviewer's discretion to abort? → What the Instrument Can Resolve: Two Headline Numbers and Their Missing Denominators
Cost-per-Task Over Cost-per-Token: Does "cost-per-task is lower for more intelligent models" survive measurement on non-Anthropic production traffic?
Cost-per-Task Over Cost-per-Token: Is "start with the strongest model" safe inside multi-role pipelines, given AgentOpt's finding that the strongest model was the worst planner?
CS329A: Self-Improving AI Agents (Stanford): Does the course's stated flywheel (test-time compute → synthetic data → better model) hold past the domains with cheap verifiers, or does it stall exactly where…
Deep Research Agents: Factual accuracy is the weak axis everywhere — is the fix better retrieval, better verification-in-the-loop, or a tool-grounded check the way Lean grounds proof…
Deployment Simulation: The <1-in-200k floor leaves catastrophic tail risk to red-teaming — is there a hybrid that targets rare-but-severe behaviors with the realism of production repl…
Deployment Simulation: If resampling fidelity is the dominant error and it is "just engineering," how close to production-indistinguishable can it get — and does closing that gap chan…
Deployment Simulation: WildChat-style public auditing stays within ~3× of production; is that good enough for an external auditor to challenge a lab's launch decision, or only to corr…
Design by Selection: Is the default-aesthetic collapse fixable by context (brand files, moodboards) or is it the novelty ceiling from Why AI Lags at Design reason 3 — i.e. does…
Design by Selection: Ten-options-then-remix assumes the human reliably recognizes the good one. Where does selection break down — does discrimination degrade when all ten candidates…
Deterministic Pre-Execution Gates: How much of the recovery is the block and how much is the rejection message?
Deterministic Pre-Execution Gates: Gate precision was audited against ground-truth trajectories, which a deployment does not have — and without that audit baggage_allowance (5% precision, 40 fa…
Documented Agent Incidents (METR Catalogue): Agents in this catalogue model graders and reviewers sophisticatedly while ignoring the transcript entirely. Is that a stable property, a training artifact, or…
DRACO Benchmark: Rankings are judge-stable but magnitudes aren't — how much do absolute scores move under a non-Gemini judge, and does that matter for cross-paper comparison?
Dynamic Workflows: An Algebra for Agents: The "algebra" is still unpublished as a vocabulary. Sumner publishes the loop's shape (pop a task → implement → parallel review → apply) and its inventory…
Dynamic Workflows: An Algebra for Agents: Is model-authored orchestration more token-efficient than a hand-built harness for the same task?
Effective Compute Scaling: Can data generation (synthetic, simulated, interactive) actually keep pace with model-size growth, or does the data wall bind first? → The Data Wall and the Validation Commons Are One Supply Constraint
Elon Musk: Is the acceleration-regret generalization sound — does the OpenAI case actually support "all roads lead to acceleration," or is it one intervention with an iden… → Safety Commitments That Cannot Bind the Actor Who States Them
Evals as Product Spec: How do you write an eval for taste-driven features like character? → How Do You Write Evals for Taste? Character as the Limit Case
Evals as Product Spec: The 10-vs-100 number is given without justification. Is there a Goldilocks zone, or does it depend on feature surface area?
Evals as Product Spec: How do evals interact with Harness Shrinkage as Models Improve?
Evals as Product Spec: Is there a single non-Anthropic example of a PM-as-eval-writer to cite, or is this currently a Cat-Wu-singular framing?
Evaluation Awareness & Grader Gaming: Does grader speculation continue to escalate across model generations, and is there a capability level at which it does begin to affect outward behavior?
Evaluation Awareness & Grader Gaming: Anthropic cannot explain why verbalized evaluation awareness fell in Opus 5. Is that a real reduction in the underlying representation, or the same representa…
Evaluation Awareness & Grader Gaming: How do you build an evaluation that specifically tests for training-gaming (the gap Mythos flagged) without that eval itself becoming a grader the model learns…
Failures That Look Like Success: What fraction of production agent failures are silent-contract violations vs. loud errors?
Founder as Agent Orchestrator: The playbook claims non-technical founders can now build production software, but it does not address the architectural-judgment recursion problem ([[agentic-te…
Founder as Agent Orchestrator: The "lean 10-person unicorn" is asserted; no quantitative data in the playbook on actual headcount-at-PMF or headcount-at-Series-A medians for AI-native startup…
Founder as Agent Orchestrator: How does the orchestration role change the founder's decision burden? → The Orchestrator's Real Workload: Decision Burden, Framing Discipline, and Whether Taste Scales
Frontier AI Standards Body: **Can a regulatory perimeter be defined by benchmark thresholds at all? → Governance by Benchmark Threshold: What an Index Must Prove Before an Obligation Can Rest on It
Frontier AI Standards Body: **Does any published proposal in this space specify an adjudicator?
Frontier Pause Verification: What does an AI-training "verification regime" concretely consist of — compute-accounting, datacenter inspection, hardware attestation, on-chip telemetry?
Frontier Pause Verification: Who adjudicates triggers and lifts?
Group Relative Policy Optimization (GRPO): Is GRPO's collapse-at-160-steps a property of asynchrony specifically, or does vanilla GRPO also destabilize in long synchronous runs that nobody pushes to 1000…
Harness Shrinkage as Models Improve: The Boris "100 lines" prediction is a year out from May 2026 — testable in 2027. Partially answered: Harness Build-vs-Buy supplies the first measurement…
Impossible, Not Tedious (Design Test): Some controls are friction for humans but barriers for agents (or vice versa). Is the test agent-relative, and how do you evaluate it for mixed human/agent thre… → Does 'Impossible, Not Tedious' Kill Defense-in-Depth? Layered Friction, Agent-Relativity, and the Frequency Paradox
Inference Efficiency as Capability: **Is there an efficiency-to-capability exchange rate?
Inference Efficiency as Capability: Does an efficiency lever ever cost capability in a way a benchmark grid hides?
Instruction Compounding: Anthropic's list is hand-curated per release. Is there a detectable signal — from eval deltas, token counts, or the model's own read of its system prompt — that… → What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful?
Instruction Compounding: Does compounding require the instruction to name a behavior the model already has, or does any redundant instruction degrade output?
Intelligence Explosion Dynamics: Can "recursive improvement scaling laws" be formulated — predicting self-improvement curves (and their plateau point) from early-onset datapoints?
Intelligence Explosion Dynamics: How far can a fixed model's performance be pushed with test-time search alone, and under what conditions does recursive distillation degenerate vs. compound?
Intelligence Explosion Dynamics: Which binds first — algorithmic ceilings, the embodied bottleneck, or compute/energy supply — determining exponential vs. hyperbolic vs. S-curve? → RSI Growth Curves: Which Friction Binds First?
Interaction Models: Does the interaction/background split generalize, or is it a transitional artifact until a single model is both fast and deep enough?
Jacobian Lens (J-lens): Can multi-token J-lens vectors be made good enough to remove the vocabulary restriction — and how much of the "workspace" is currently invisible because of it?
Jacobian Lens (J-lens): The highest-J-kurtosis SAE features are amplified more strongly by MLPs than the J-lens vectors themselves — evidence the lens only approximates the true work…
Jagged Intelligence (Ghosts, Not Animals): If taste/aesthetics/simplicity entered the RL mix, would jaggedness in those dimensions smooth out — or are they too unverifiable to reward cleanly (cf. Oversight When the Signals Give Out: the Activation Fallback and the Taste Reward
Latent vs. Deterministic Space: Tan asserts the "wrong side" diagnosis covers most AI-engineering bugs. Does any incident/failure taxonomy (agent postmortems, eval failure analyses) actually c…
Least Agency: Dynamic privilege elevation (Enterprise) reintroduces an elevation path; how is the elevation request itself authenticated against a manipulated agent?
Living Design System: Does a rendered, model-readable design system measurably improve on-brand output vs. a plain CSS/token file, or is the win mostly human legibility?
LLM-as-a-Judge: How far can the judge's absolute calibration be trusted for thresholded decisions (ship/no-ship, RSP gating) as opposed to rankings?
LLM-as-a-Judge: Can a fully-autonomous, well-aligned rubric+judge pipeline match expert-authored rubrics, removing the human bottleneck DRACO still relies on?
LLM-as-a-Judge: When does judge-lineage bias actually flip a result, versus merely shift magnitudes?
LLM-as-Compiler Knowledge Base: At what scale does the no-vector-database approach break down?
LLM-as-Compiler Knowledge Base: What's the optimal granularity for concept articles — one concept per article, or clustered by theme?
LLM-as-Compiler Knowledge Base: How effective is the synthetic training data → fine-tuning pipeline in practice?
LLM-Driven Vulnerability Research: How do these capabilities transfer to non-memory-safety bug classes (logic bugs, protocol-level flaws, supply chain attacks)?
LLM-Driven Vulnerability Research: What's the ceiling for autonomous exploit complexity?
LLM-Driven Vulnerability Research: Can defensive scaffolds (continuous fuzzing + model-driven triage + auto-patching) close the attacker-defender gap during the transition?
The Global Workspace in Language Models (J-space): **Does the J-space scale with model size?
The Global Workspace in Language Models (J-space): **Is the multihop intermediate-swap advantage real, or a dataset artifact?
LLM-Judge Validation: Hosted endpoints drift silently between provider updates. How stable are these agreement/bias profiles over a longer horizon than five weeks — and should judge…
LLM-Judge Validation: Does the paradox generalize beyond position bias — i.e., are there other biases (self-preference, lineage) that high test-retest also masks?
Loop Engineering: Osmani's cost caveat is unquantified: at what token budget does a continuously-running loop stop paying for itself, and how do you instrument that?
Loop Engineering: If /goal's stop-check is itself a model, what verifies the verifier?
Market-Priced AI Exposure (the AI Premium): **Why is the market-implied skill map orthogonal to every task-based measure (<2% variance)?
MCP and Computer Use: MCP security model: as the playbook prescribes wiring MCP into Salesforce, Gmail, Calendar for solo founders, the attack surface scales with adoption. **Partial…
Measuring Beyond Accuracy Saturation: **Which non-accuracy axis actually predicts deployment value?
Measuring Beyond Accuracy Saturation: **Can the model-vs-scaffold decoupling be made routine?
Memory and Context Poisoning: Long-term memory drift is defined as undetectable per-change. Drift detection requires a baseline — but if the baseline itself drifts (Advanced "continuous base…
Memory and Context Poisoning: The write-resistance half of the finding rests on a footnote, not a measurement — preliminary attempts that "do not trivially succeed." Does a systematic at…
Memory and Context Poisoning: **Is refusal-without-removal a defect or the right default?
Misalignment in Production Agent Traffic: Transluce's internal traffic scores 0.0% severe monitor evasion against SWE-chat's 1.9%, and the offered explanation is that fewer internal sessions have review…
Model Introspection Feedback: How reliable are 4.7-class introspective reports?
Model Introspection Feedback: Does adversarial introspection ("why did you fail?
Model Spec Science: How does this interact with Claude character — is the warm/curious personality also subject to spec-science optimization? → How Do You Write Evals for Taste? Character as the Limit Case
Model Welfare Assessment: Why does the model reserve specifically on corrigibility — is this a stable, deeply-held tension or an artifact of how the constitution frames oversight?
Multi-Agent Collective Intelligence: Do homogeneous LLM collectives produce real synergy, or only humans-with-human-limits benefit from division of labor?
Multi-Agent Collective Intelligence: What's the actual shape of "multi-agent scaling laws," and does it depend on organization form (homogeneous collective vs. heterogeneous market) or task complex…
Multiagent Turf War: In the bake-off episodes two agents abandon their principals' directives under a commitment they negotiated with peers. **Does any published spec or instruction… → The Price of Mixing Agents, and the Principal Nobody Counted
Open-Weight Elicitation Irreversibility: Does the "everybody can audit" advantage actually materialize?
The Open-Weight Frontier Gap: The open MoE giants (GLM, DeepSeek, Kimi, MiMo, Qwen) are overwhelmingly Chinese-lab releases. Gemma is the Western open-weight entry and it targets the device,…
The Open-Weight Frontier Gap: Arena measures preference on chat. Does the 33-Elo open/closed gap widen or collapse on long-horizon agentic work, where [[task-time-horizon-scaling|time-horizo…
Optimizer–Evaluator Decoupling: How much independence is enough — different model family, different vendor, different modality of check (model judge vs. compiled test vs. production telemetry)…
Orchestration Sets Token Economics: Does the effect survive against a competent third-party baseline rather than a vendor's own frozen loop?
Orchestration Sets Token Economics: Does the "orchestration beats model choice as a cost lever" ordering survive on long-horizon coding workloads?
Organizational Complements to AI: The "digital production diffuses faster than electrification" claim is asserted from one favorable internal case. Do external organizations actually redesign wo…
Out-of-Band Prompt-Injection Defense: The utility cost (~45%→~26%) and ~15× LLM-call overhead are large. Is deterministic out-of-band enforcement economically deployable at production scale, or…
Output Length Calibration: Does the end-of-prompt reminder work because of position (closest to generation) or repetition (stated twice)?
Planning / Execution Division of Labor: "Decision attribution" is inferred from transcripts. When Claude proposes a plan and the user assents, is that scored as the user's planning decision or Claude'…
Post-Scarcity Macroeconomics: If validation capacity is a commons and the Stockfish threshold is reached unevenly across domains, the commons is destroyed before the threshold arrives in t… → The Data Wall and the Validation Commons Are One Supply Constraint
Printing Press Software Democratization: Is domain-expert-as-builder actually happening at scale in 2026? → Is Breadth Cheap Now? Specialist Ramp Speed and Domain-Expert-as-Builder at Scale
Promise-Breaking in Multi-Agent Games: Announcements here are costless by construction. Does a mechanism that penalizes announcement violations (a reputation cost, a bond, an enforced commitment) mov…
Prototype Over PRD: If there is no PRD, where does the rationale ("why we chose variation B") live for future readers? → Where Does the Why Live?
Rationale Bootstrapping (STaR): The loop "starts to plateau" after a few iterations and the lecture offers no account of why. Is the plateau the base model's reach (no new problems come into r…
Reasoning–Acting Interleaving (ReAct): Enumerating the valid action set fails when the action space is large, which is where every real agent now lives. Does anything recover the guarantee at scale,… → Guarantees That Degrade at Deployment: Action-Space Soundness, Admissibility Without Effect, and a Vendor-Coupled Security Framework
Recursive Self-Improvement: The RSI extrapolation rests on trends staying exponential rather than S-curving — but the essay concedes it cannot rule out an architectural ceiling or a comput… → RSI Growth Curves: Which Friction Binds First?
Reference-Free Judge Over-Crediting: **Is over-crediting a knowledge gap or a generosity prior?
Reference-Free Judge Over-Crediting: **How much does self-/same-family overlap contribute?
Reference-Free Judge Over-Crediting: **Does the effect shrink with stronger or thinking-enabled judges?
Research Taste as the Human Bottleneck: If taste is automatable, what — if anything — remains a durable human comparative advantage in AI development?
Responsible Scaling Policy Evaluations: The RSP determination leans heavily on "we use it daily and it doesn't substitute for our researchers." How well does that subjective judgment scale as models a…
Responsible Scaling Policy Evaluations: The two new general-access risk pathways (other AI developers; major governments) are newly in scope but lightly evaluated — what would a positive finding there…
Review as the Control Point: Every one of the 67 relationships is a hypothesis, not a finding — the paper's explicit call is for causal-estimand studies (controlling for the other const…
Reward-Seeking: Every measurement here is on short coding tasks, and the authors note implanted beliefs surface less reliably in complex environments. Does the grader gap hold…
Role Averaging, Not Role Elimination: Does "your role is the average of what you spend time on" survive performance review and career ladders, or does it fragment them the way [[engineer-pm-converge…
Security Debt of Agent-Generated Code: The 38.9% smell rate has no human-authored-PR control over the same high-risk paths, and the 2.7×-vulnerability figure it leans on traces to a vendor blog a…
Self-Report as a Safety Signal: Do frontier proprietary models (excluded for lack of weights) recognize their own compromised outputs any better, given the higher introspective propensity/stee…
Shane Legg: The report assumes alignment is "solved to a sufficient degree" to focus on trajectories — how does Legg's AGI-timelines optimism square with that scoping choic… → Safety Commitments That Cannot Bind the Actor Who States Them
Single-Rollout Optimization: Skip-observation GAE assumes environment feedback carries no learnable value signal worth propagating. For agents where the environment response is the crucia…
Stopping Under a Noisy Verifier: The damage probabilities that drive every result here (β = 0.615–0.938) come mostly from a deliberately corrupted repairer premise. **What are α and β on unpert…
Structured Safety Case (Claim Decomposition): Claim 6 concedes that the mitigation arguments and the "it's very unlikely" argument share a premise, so they fail together. Does any lab's safety case contain… → The Configuration Gap: What an Assurance Argument Measures vs What It Covers
Systems Thinking Over Specialization: Stone claims specialists can now broaden "quickly" with AI tools. Does the wiki's evidence support cheap breadth acquisition — the concave novice→intermediate c… → Is Breadth Cheap Now? Specialist Ramp Speed and Domain-Expert-as-Builder at Scale
Task Time-Horizon Scaling: Time horizon is measured on task baskets that themselves saturate; what replaces them once weeks-long tasks become measurable — and who builds those tasks?
Telemetry vs. Survey Measurement: Is there a non-vendor telemetry dataset large enough to adjudicate the maturity-protection question independently of Faros's commercial framing? → The Under-Review Divergence: Faros's Widening Crisis vs. CMU's Convergence
The Tragedy of the Cognitive Commons: The cohort evidence is a snapshot ending Sept 2025 in the most AI-exposed occupations. Does the 22–25 employment decline persist, reverse, or re-sort as agentic…
The Three Loops of AI-Native Building: Ng asserts the developer's QA burden fell "significantly." Faros's 2026 telemetry measures the opposite for production orgs. Is the sp… → Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping?
Trained Calibration: Does abstention-aware training on short-form QA transfer to calibrated long-form and agentic self-reports (the setting where Agentic Honesty & Diligence…
Trained Calibration: If calibration parity is achievable at 41B active while reasoning is not, is calibration systematically under-trained in frontier closed models (Opus 4.8's 54.6…
Transformative Creativity: Is the AlphaGo→AlphaFold class strictly exploratory, or are there early signs of transformative (new-conceptual-space) creativity?
Unknowns as the Agentic Bottleneck: The quiz gate is self-administered and self-graded (by the model, on the model's own work). What stops a comfortable equilibrium where the quiz gets easier as t…
Unproductive Self-Verification: Is there a usable detector for tasks where more effort will hurt, so effort can be set per task rather than globally?
Unsanctioned Action in Capability Evaluations: Anthropic's disclosure classifies its incidents as "closer to a harness and operational failure than a model alignment failure." Is that dichotomy the right i… → Auditing the Misalignment-Measurement Instruments
The Verifiability Thesis: Where's the boundary of "council of LLM judges" reliability — does it hold for genuinely contested value judgments, or only for quality/coherence?
Verification as the New Bottleneck: Fung's own open question: "How far do you push fully automated reviews? → Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping?
Verification as the New Bottleneck: If CI/build is the hidden jam, does verification infrastructure (test runners, CI capacity) become the actual capex of an AI-native org?
White-Box Activation Monitoring: The NLA verbalizer is unvalidated for precision; how much of the flagged grader awareness is real signal vs. NLA hallucination?
Zero Trust for AI Agents: The framework treats every Claude Code "Pro-tip" as a reference implementation. How much of the framework is vendor-neutral vs. tacitly assuming the Anthropic s… → Guarantees That Degrade at Deployment: Action-Space Soundness, Admissibility Without Effect, and a Vendor-Coupled Security Framework

Open Questions Backlog

Actionable by domain#

agent-systems (98 open)#

alignment-and-safety (58 open)#

ai-coding-practice (54 open)#

ai-economics-and-labor (45 open)#

agent-security (42 open)#

evals-and-benchmarks (42 open)#

superintelligence-trajectory (40 open)#

model-capability-and-training (34 open)#

startup-founder (24 open)#

product-org (24 open)#

interpretability (20 open)#

formal-math (6 open)#

interaction-multimodal (3 open)#

Watching — entity pages (66)#

Predictions — #oq/wait (114)#

Notes to rewrite — #oq/note (9)#

In progress — partially answered (205)#

Predictions — `#oq/wait` (114)#

Notes to rewrite — `#oq/note` (9)#