Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping?

The questions#

Five related #oq/now items from the ai-coding-practice oversight cluster, answered as one synthesis:

AI as Primary Author — what does "acceptance" mean when the agent applies the change directly and the human's acceptance is not reverting it?
Compute Allocator — does the allocator framing risk the oversight-fatigue / rubber-stamping failure modes, where the human nominally decides but actually rubber-stamps?
Verification as the New Bottleneck — how far do you push fully automated reviews?
Telemetry vs. Survey Measurement — is the Faros–DORA "contradiction" partly a category error, both true at their own layer?
The Three Loops of AI-Native Building — is the Ng-vs-Faros QA-burden split really 0-to-1-vs-production, or optimism bias in self-report?

Answer 1: "Acceptance" is three constructs wearing one number#

Faros's 60% acceptance figure aggregates acts with entirely different evidentiary weight (AI as Primary Author). Split by who acts and who looks:

Affirmative adoption — a human reads a suggestion and applies it (the autocomplete-era meaning). Acceptance is a decision.
Reviewed non-reversion — the agent applies the diff; an independent human examines it before merge. Acceptance is delegated authorship plus a second set of eyes.
Bare non-reversion — the agent applies; nobody independently looks; "acceptance" is the absence of an intervention. This is the rubber-stamp class, and the only one that deserves the alarm attached to the aggregate.

The construct instability is measured, not hypothetical: on GitHub, agent PRs are most often examined only by the developer who invoked the agent (author-only review 40.1% vs 21.5% for human PRs), and whether that developer counts as an independent reviewer (agent-as-author) or a self-reviewing author (agent-as-tool) flips the sign of the independent-review trend on the same rows (Review as the Control Point). The Faros-vs-CMU "divergence" on under-review dissolves the same way once metric, population, time axis, and authorship unit are aligned (The Under-Review Divergence: Faros's Widening Crisis vs. CMU's Convergence) — the datasets never disagreed, the constructs did. What the two tell us jointly about the rubber-stamp class: it is large in enterprise snapshots (Faros: 31.3% of PRs merged with no review — Acceleration Whiplash) and transient in open source (CMU: merged-agent-PRs-with-no-review falls from >50% mid-2025 toward the ~14% human baseline by early 2026). Whether enterprise follows the open-source convergence is the live empirical question, not answerable from either dataset alone.

Practical rule extracted: stop reporting "acceptance rate"; report the three-way partition, and treat only bare non-reversion as the oversight-erosion signal. A rising share of category 2 is delegation working; a rising share of category 3 is the control failing.

Answer 2: Yes — the allocator's central failure mode, documented at three evidence layers#

The Compute Allocator framing presumes the scarce human input is judgment quality, while the behavior it celebrates — 99% disposable scaffolding, hundreds of concurrent agents — maximizes exactly the volume that degrades judgment. The rubber-stamp risk is not speculative; the wiki holds it at three independent evidence layers:

Controlled/experimental: oversight fatigue raises minor errors +11% and major errors +39%; the employee framing swaps fatigue for under-engagement (an 18% drop in error catching) — two failure surfaces, no free framing (AI Brain Fry). And self-report is blind to the damage: heavy delegators report no learning deficit (The Automation–Optimism Link) while randomized measurement shows automation-mode gains vanish once the AI is removed (Experimental Learning Impact of Generative AI).
Telemetry: 31.3% of PRs merged with no review, review time up ~5×, daily PR contexts per developer +67.4% (Acceleration Whiplash).
Practitioner discourse, mechanism-mapped: load lowers review depth and motivation (P1–P3: "skimming, not reading"), and surface plausibility disarms the reviewer (P4: polish lowers the guard — "seniors rubber-stamping code that looks idiomatic but hides subtle bugs"), compounding into comprehension debt even when depth looks high (P10/P11: "we are not reviewing the recipe, we are tasting the dish") (Review as the Control Point).

Two aggravators are specific to the allocator role: allocation quality has no feedback loop (the page's own open question — nothing tells an allocator they spent compute badly rather than merely a lot), and the planning/execution division (Planning / Execution Division of Labor) means the human's retained 70% share is planning decisions — precisely the decisions whose quality degrades invisibly, since a rubber-stamped plan approval is transcript-indistinguishable from a considered one (that page's own rubber-stamping measurement caveat).

But the failure mode is a threshold, not a destiny — the countermeasures are also in evidence. What survives brain-fry volume is review redesign, not review effort: sample-based depth instead of every-output review, concentration on high-stakes decision points (AI Brain Fry); a merge gate contingent on demonstrated understanding rather than signature (the quiz gate — Unknowns as the Agentic Bottleneck); risk-tiered governance that gates only material changes (P17); and right-timed, right-channel participation rather than more participation — HAS-Bench finds human input is configuration-dependent and non-monotonic, with over-intervention breaking previously-solved tasks (Configurable Human Participation). So: yes, the allocator framing carries the rubber-stamp risk as its central failure mode; it remains a real control only where the merge condition is understanding, sampled deeply, not presence, spread thin.

Answer 3: "How far" is a partition, not a dial#

Fung's question assumes a single automation frontier to push. The evidence decomposes it (Verification as the New Bottleneck, Review as the Control Point):

Automate fully: mechanical verification. Style, lint, obvious bugs, spec-drift against a checked-in spec, tests — where the verifier is deterministic or near-deterministic, automation is pure win and already the practice on the Claude Code team.
Contested and unmeasured: automated quality/security judgment. P9 is genuinely contested — one camp cites ~82% planted-bug catch rates, the other "catches style, misses the race-condition-under-load kind of problems" and manufactures false confidence. The vendor's own precision/recall claim for Opus 5 review is unmeasured first-party positioning and doesn't settle it. Full automation reliably buys throughput and latency (P8); nobody has shown it buys quality.
Never fully automatable on current evidence: the functions review performs besides defect-catching. Human review depth is how reviewer skill grows (P12/P13: "how can you review what you don't yet know how to build?"), how collective ownership and knowledge transfer happen (P14: "when no human has deeply reviewed the logic, nobody owns it"), and how comprehension debt is paid down rather than accrued (P15). These erode under automation even if the automated reviewer catches every bug — the binding constraint on "how far" is not defect detection but ownership and skill reproduction.

So the answer has a shape: push automated review to 100% of mechanical checking; keep human depth on a risk-tiered, sampled slice sized to preserve comprehension and skill, not to re-check the machine. The frontier's position is set by the theory's moderators (automated-reviewer capability, process calibration) — which is why it has no fixed value and the moderator thresholds remain the open measurement question.

Answer 4: Partly a category error — but not entirely#

Yes, the clean half of the Faros–DORA conflict is a category error. Surveys measure felt productivity (real: individual task completion genuinely rose — the feeling isn't wrong), telemetry measures system outcomes (queues, incidents, review latency) that haven't yet propagated into feeling; during a fast transition the two layers legitimately diverge, which is Faros's own "perception lags reality" mechanism read symmetrically rather than as a winner-picker (Telemetry vs. Survey Measurement). The AEI's linked design (telemetry joined to surveys per person) already treats them as complements, and the CMU study shows the sharpest reason neither instrument "wins": even non-vendor telemetry is direction-unstable without a causal model of what review is for (Review as the Control Point — Pearl's "data are profoundly dumb"). Two instruments, two layers, both true — most of the "contradiction" dissolves.

The residual that is not a category error: the maturity-protection claim. DORA says strong foundations protect; Faros says its telemetry shows they don't. That is one proposition about one layer (system outcomes), and it stays substantively contested — Faros's version aligns with its vendor incentive, CMU's moderator theory argues the sign is team-set (closer to DORA) but measures no maturity effect. Category-error dissolution cleans up the debate's framing; it does not settle its one real disagreement.

Answer 5: Both-and — the scope split is real, and the optimism bias is unexcluded#

The Ng-vs-Faros disagreement (The Three Loops of AI-Native Building vs Acceleration Whiplash) does not have to pick one explanation:

The scope split does most of the work. In 0-to-1 personal builds, the external cost of a wrong build is near zero, the agent's self-tests are an adequate verifier, and there is no review queue, no incident budget, no maintainer who must comprehend the diff in a year. The functions that make review expensive in production (Answer 3's ownership/comprehension functions) barely exist, so the QA burden genuinely can fall. Ng's "promoted out of QA" is plausible for his regime.
The optimism bias is simultaneously documented and applies to exactly this kind of claim. Ng's evidence is self-reported felt burden — the instrument class that Answer 4 shows lags system reality, that reports no learning deficit where randomized measurement finds vanished gains (The Automation–Optimism Link, Experimental Learning Impact of Generative AI), and that feels productive while error rates climb (AI Brain Fry). "Significantly decreased" from self-report is a feeling with a known rosy skew, not a measurement.

So the either/or dissolves: the scope difference explains why Ng's burden could truly be lower; the bias literature explains why his report can't quantify it. The standing tiebreak — Faros's telemetry outranks both practitioner opinions for the org case — is unchanged, and CMU's convergence finding adds the one hopeful org-side datum: teams learn to review agent code, so the burden trend is not monotonic doom. What's missing and would settle it: any measured QA-time series for 0-to-1 builders, which no source yet provides.

Addendum (2026-07-29): a fourth evidence layer, and it lands on Answers 2 and 3#

Security Debt of Agent-Generated Code (Sakib et al., arXiv 2607.12428, empirical) was compiled after this synthesis and changes two of its answers at the margin — not by contradicting them, but by supplying the outcome-level layer the argument was missing.

Answer 2 gains a fourth layer. The three layers cited above are experimental (error rates), telemetry (no-review counts), and discourse (mechanism). All three measure the conditions for rubber-stamping. This one measures a consequence on a specific artifact: of 74 genuine live credentials in 4,022 agentic PRs, 81.1% reached integration with no comment from any bot or human reviewer, and humans — not agents — committed 67.6% of them. The rubber-stamp risk is no longer only inferred from load and error-rate proxies. (Construct caveat: the paper operationalizes reviewer detection as "a reviewer commented," and 60 of the uncommented secrets were nonetheless removed; the defensible claim is "no comment before integration," not "undetected.")
Answer 3's partition survives but its first clause gets demoted from description to prescription. "Automate 100% of mechanical checking" reads as already-solved. Hard-coded credentials are the most mechanically detectable smell class there is, seven distinct commercial detectors appear across the sample, and they collectively commented on 18.9% of live credentials. Mechanical verification is the part we can automate; it is not the part we have automated.
Answer 1's convergence hope needs a qualifier. The three-way acceptance partition treats "reviewed non-reversion" (category 2) as delegation working. This source shows category 2 can contain an unreviewed-in-substance PR: coverage convergence is not efficacy convergence, and a PR with a reviewer attached still merged live credentials four times in five.

Addendum (2026-07-29): a deployed calibration of Answer 3, and a sharper reading of Answer 1#

Risk-Tiered Auto-Approval (PostHog's StampHog, case-study) is the first production instance in the vault of the partition Answer 3 prescribes, and it lands slightly off where the prescription pointed.

Answer 3's frontier is drawn by check kind, not by check quality. The partition splits "mechanical verification" from "quality/security judgment" — a distinction about what is being checked. StampHog's split is about what does the checking: three deterministic gates (PR state, a blast-radius keyword deny-list, a <500-line/<20-file ceiling) decide, and the LLM runs last, permitted only to tighten. That is a stricter line than Answer 3 drew, and a defensible response to the same evidence — since the layer whose judgment Answer 3 marks contested is exactly the layer this design refuses to let decide.
Answer 1 gains a fourth category, and it is an improvement in accounting. The three-way acceptance partition has no slot for explicit, criteria-published, logged non-review. What StampHog displaced was a Slack stamp exchange in which an uninvolved engineer approved "a change they had little to no context on" — already bare non-reversion wearing a signature. Automating it makes the unreviewed set enumerable and its criteria auditable, which is strictly better accounting without being better oversight. The corollary for reading such numbers: "an agent approved 1 in 3 merged PRs" is not "a third of review was automated away," because a large share of that third was ceremony.
What the case study cannot supply. No false-approval rate, no escaped-defect count. Answer 3's contested zone (P9) stays contested; this adds scale to the deployment evidence, not measurement. The checkable test is sitting in PostHog's own history — auto-approved PRs versus the human-stamped baseline the channel used to produce.

One consolidated takeaway#

Across all five questions the same three-step pattern resolves the debate: (1) split the construct (acceptance into three acts; review into defect-catching vs skill/ownership reproduction; "the contradiction" into layers; the either/or into mechanism + magnitude), (2) weight instruments by what they can see (self-report sees feeling, telemetry sees behavior, neither sees why without a causal model), and (3) relocate the control: human oversight of AI-authored code remains a real control only where it is redesigned — three-way acceptance accounting, understanding-gated merges, sampled depth on a risk-tiered slice, automation owning everything mechanical. Review effort spread uniformly across agent-scale volume is already rubber-stamping wearing review's badge.