AI Coding Practice, in order.

Open questions 50 open

• Acceleration Whiplash
  • Faros's own deferred question: do the bug/incident increases persist when **normalized for PR size**, or do larger PRs account for most of the quality deterioration? (If the latter, hard PR-size limits are the highest-leverage fix.)
  • Code churn +861% is genuinely ambiguous (Faros lists three explanations: rework of AI code, productive legacy refactoring, or accelerated polish). The cross-customer metric can't resolve it — a real gap, not a finding.
• Agentic Coding Work-Composition Shift
  • The window is seven months and the value proxy is coarse/relative. How much of the +27% is genuine task-complexity growth vs. classifier/marketplace-matching drift?
  • The study excludes headless/SDK/IDE usage — a "substantial share," and likely the most automated/end-to-end. Does including it accelerate or reverse the composition shift?
  • If "fixing" keeps falling, is that because models break less, or because broken-code work is migrating to non-interactive pipelines this study doesn't see?
• Agentic Work Systematization
  • Custom skills encode org-specific context — but who maintains them as the codebase and conventions drift? Systematization could itself become a debt surface ([[agentic-technical-debt]]) if skills rot.
  • Does systematization *cause* deeper delegation or merely *correlate* with already-intensive users? The paper shows the association, not the direction.
• AI as Primary Author
  • The 60% figure aggregates very different tools and modes (autocomplete acceptance vs. agent-applied diffs). What does "acceptance" mean when the agent applies the change directly and the human's "acceptance" is not reverting it? **Sharpened** by [[review-as-the-control-point]]: on GitHub, agent PRs are most often examined *only by the developer who invoked the agent* (author-only review 40.1% vs 21.5% for human PRs). Whether that counts as review at all is a definitional choice (agent-as-author ⇒ a second set of eyes; agent-as-tool ⇒ self-review) that literally flips the sign of the trend — so "acceptance" and "review" blur into the same unresolved construct.
• Building Is Cheap, Arguing Is Expensive
  • When does "generate three and compare" become wasteful — at what decision weight is a real argument (or a design doc) still cheaper than three implementations?
  • If design discussion lives in PRs/prototypes, where is the *rationale* recorded for future readers — does the "why we chose this" knowledge survive, or does it share the staleness problem of [[code-as-source-of-truth]]?
• Code as Source of Truth
  • What knowledge genuinely *can't* live in the codebase (org strategy, the "why," cross-team context) and therefore still needs a durable doc — and how do you keep that small slice current?
  • If onboarding is "ask Claude," what happens to the tacit knowledge that was previously transferred socially in deep-dives — is it captured anywhere, or quietly lost?
• Compute Allocator
  • Is 1% a Thariq-specific number or a regime? For larger, more code-heavy projects the production residue is presumably higher; what sets the ratio?
  • Allocation quality is hard to measure — what's the feedback loop that tells an allocator they spent compute *badly* (vs. just spending a lot)?
  • Does treating humans as "compute allocators" risk the [[ai-brain-fry|oversight-fatigue]] / [[human-ai-accountability-redesign|accountability]] failure modes the HBR research flags, where the human nominally decides but actually rubber-stamps?
• Configurable Human Participation
  • The "human" is an LLM simulator (GPT-4.1) that also judges — how much of the configuration-dependent structure is a property of *human-agent collaboration* versus an artifact of GPT-4.1 modeling both sides? The Appendix C.6 simulator-swap is the only check, on a subset.
  • A4 is instantiated as a *fixed one-shot* proactive intervention. Real proactive humans time their input adaptively — does the "premature/distracting intervention breaks tasks" result hold, worsen, or vanish under a human who chooses *when* to interject?
  • The optimal channel is shown to be pattern-specific, but pattern is an oracle label assigned at construction. Can an agent *infer* which pattern it is in (and thus which channel to solicit) at runtime — the actual deployed skill the paper says agents lack?
• Design Concept Grilling
  • How does grilling change for team work where multiple humans need to align? Pocock's hint: pair-program with the agent in the room, treat it as a third interlocutor.
• Disposable Micro-Apps
  • Where's the line between a disposable micro-app and tool sprawl? If every edit spawns a bespoke UI, does the workflow fragment?
  • Does the copy-back-to-markdown round-trip generalize beyond config-shaped data (rules, tables) to richer artifacts?
  • Could these micro-apps be templated/reused rather than regenerated — and at what point does that defeat the "disposable" framing and turn into [[living-design-system|durable tooling]]?
• HTML as the New Markdown
  • HTML is heavier to diff and version than markdown — what happens to plan history and review when artifacts are single-file websites? ([[disposable-micro-apps]] copy-back-to-markdown is one patch.)
  • Does this generalize past one expert practitioner, or does it require Thariq-level fluency with Claude to be worth the overhead?
• Living Design System
  • How does the `design_system.html` stay in sync as the codebase evolves — re-extract on a cadence, or wire it into CI?
  • Does a rendered, model-readable design system measurably improve on-brand output vs. a plain CSS/token file, or is the win mostly human legibility?
  • At what project size does maintaining the artifact cost more than the consistency it buys?
• LLM-Assisted Grey-Literature Theory Building
  • The authors couldn't locate the **saturation point** because synthesis was manual — how few documents actually suffice, and can a cheaper sample match the 3,100-doc theory?
  • Automating the **codes→theory** step failed with a naïve bottom-up prompt; is that a prompt/scaffolding limitation or a genuine ceiling on LLM interpretive synthesis over thousands of codes?
  • The three-lens design manages coder bias, but the **relevance judge and segmenter are single-model** — do those upstream gates impose their own systematic slant on what reaches the codebook?
• Outsource Your Thinking, Not Your Understanding
  • If understanding is the bottleneck, is the highest-ROI skill *learning how to build understanding fast* (knowledge-base hygiene, asking the right projections) — and can that be taught?
• Planning / Execution Division of Labor
  • "Decision attribution" is inferred from transcripts. When Claude proposes a plan and the user assents, is that scored as the user's planning decision or Claude's? The rubber-stamping boundary is exactly where the measure is hardest.
  • Headless/SDK/pipeline usage (excluded here) is where execution autonomy is highest and planning is front-loaded into a single prompt — does the 70/20 split survive there, or collapse toward full delegation?
• Review as the Control Point
  • Every one of the 67 relationships is a **hypothesis, not a finding** — the paper's explicit call is for causal-estimand studies (controlling for the other constructs) to confirm, reverse, or drop each edge. Which of P1–P17 survive measurement?
  • The paper's own question: *which decisions, under which conditions, push the system toward the virtuous loop rather than the vicious one?* — the system-dynamics leverage-point analysis it gestures at but doesn't run.
  • The three contested edges (automated review → quality/security, P9; governance → latency, P17; and one more) are contested *because* their sign is moderator-set — what are the moderator thresholds that flip them?
• Telemetry vs. Survey Measurement
  • Surveys and telemetry measure different things (felt productivity vs. system outcomes); is the "contradiction" partly a category error — both true at their own layer — rather than one being wrong?
• The Three Loops of AI-Native Building
  • Ng asserts the developer's QA burden fell "significantly." [[acceleration-whiplash|Faros's 2026 telemetry]] measures the opposite for production orgs. Is the split really 0-to-1-vs-production, or is Ng's self-report subject to the same [[automation-optimism-link|optimism bias]] the survey literature keeps finding?
  • The external loop is the unshortened one. Is that physics (users take time to react) or an unautomated frontier (synthetic users, [[deployment-simulation|deployment simulation]] applied to products rather than models)?
• The Verifiability Thesis
  • Where's the boundary of "council of LLM judges" reliability — does it hold for genuinely contested value judgments, or only for quality/coherence?
  • The "labs care" dependency is fragile: capabilities can appear or stagnate based on lab priorities you don't control. How should a product hedge against the data-distribution rug-pull?
• Unknowns as the Agentic Bottleneck
  • Is "the first model bottlenecked by my unknowns" a property of Fable or of Thariq? A frontier-lab engineer with deep model fluency hits the human-side ceiling before an average user does — which would make this a leading indicator rather than a current universal.
  • The quiz gate is self-administered and self-graded (by the model, on the model's own work). What stops a comfortable equilibrium where the quiz gets easier as the reviewer gets lazier? Cf. the maker/checker problem in [[verification-as-the-new-bottleneck]].
  • Elicitation has a cost. Every technique here spends a session's worth of tokens and attention on *not building*. Nothing in the source bounds when the blindspot pass costs more than the bug it prevents.
  • If unknown knowns are extractable, are they extractable *once*? Does a codified blindspot pass become a skill file that permanently narrows the gap, or does each new territory reopen it?
• Verification as the New Bottleneck
  • Fung's own open question: "How far do you push fully automated reviews?" — where's the speed/safety balance, and how do you keep humans confident without re-introducing the review bottleneck? **Sharpened** by [[review-as-the-control-point]]: full automation reliably raises review throughput and cuts latency (its P8), but its effect on *code quality and security is contested* (P9), and the latency effect of a review-governance policy flips sign by calibration — a risk-tiered policy that gates only material changes lowers latency, a blanket policy raises it (P17). So "how far" has no single answer: the safe frontier is set by automated-reviewer capability and process design (two of that page's three moderators), not by a fixed dial.
  • If CI/build is the hidden jam, does verification infrastructure (test runners, CI capacity) become the actual capex of an AI-native org?
• Vertical Slice Tracer Bullets
  • Can the planner agent be trusted to slice vertically once told to, or does it need a verifier that flags horizontal slices? Pocock's experience: it needs the verifier, at least through 4.7.
  • How should slice granularity be tuned? Too thin = many merge conflicts; too thick = back to horizontal.
• Vibe Coding vs. Agentic Engineering
  • Karpathy hints at "one domain that's very [valuable]" for founders but won't say which (didn't want to "vague-post on stage"). What verifiable RL-environment domain is he gesturing at?