Risk-Tiered Auto-Approval

Sources#

• Deploying AI from pilot to production: A practical blueprint for CIOs and technical leaders
• State of AI in the SOC 2026: 8 Key Takeaways
• Stop being the code review bottleneck
• The State of AI Impact in Engineering: Q2 2026

Summary#

Risk-tiered auto-approval is a merge gate that decides whether a human needs to look at this PR at all, using cheap structural properties of the diff rather than a judgment about its correctness. PostHog's implementation — StampHog, described by Jina Yoon in the PostHog newsletter, July 2026 — is the vault's first documented production instance at scale: it gave the final approval on roughly 1 in 3 PRs merged into PostHog's main repo over a quarter, and handled 1.6K PRs in one month.

It is the deployed form of the construct CMU's review theory names in P17: a risk-tiered governance policy that gates only material changes lowers review latency, where a blanket policy that gates everything raises it. That page argues the moderator exists; this source shows one calibration of it running against a real merge queue.

Evidence note. case-study — a first-party account of the team's own tooling, written for a newsletter that promotes the practice. The volume figures (1 in 3, 1.6K/month) are the kind of thing a team can count reliably and there is no obvious incentive to inflate them. What the account never reports is the number that matters for safety: no false-approval rate, no escaped-defect count, no incident attribution. Treat the mechanism as documented and the outcome as unmeasured.

The gate stack: deterministic first, model last#

An engineer opts a PR in by adding a stamphog label. Four checks run:

1. PR state — no merge conflicts, no outstanding changes-requested review.
2. Blast radius — a deny-list of keywords (auth, secrets, billing, public APIs, …). Any hit routes to a human.
3. Diff size — under 500 lines and 20 files.
4. A simple LLM check — "for basic showstoppers."

The ordering is the design. Three of the four gates are deterministic properties of the diff and the repo; the model runs last, on a change that has already been established to be small and to touch nothing on the sensitive list. This inverts the default shape of AI code review, where a model is the detector and its judgment is what stands between the diff and the merge — the arrangement whose quality effect P9 marks as genuinely contested. Here the model is a veto on an already-narrowed set, not the thing doing the narrowing. It is the same structural move "prevent, don't detect" makes in the security register, arriving independently from the throughput side.

Output is deliberately minimal on both branches: approval is a bare GitHub approval with no line comments (it is explicitly not pretending to be a review), and refusal is a 1–2-sentence reason, a risk-level rating, and next steps.

Three invariants#

PostHog's own porting prompt states the safety contract it wants preserved verbatim: "fail closed, never request changes or merge, LLM can tighten gates but never loosen."

• Fail closed — ambiguity resolves toward the human, so the failure mode of a broken gate is added latency rather than an unreviewed merge.
• Never merge, never request changes — the agent's authority is bounded to one action, the approval. It cannot land code and it cannot block a human's PR with a changes-requested review. Compare Claude Code Auto Mode, where the classifier's authority is likewise one-directional (it may block a tool call, never unblock a prohibited one).
• The model may tighten but never loosen — the asymmetry that keeps a persuadable component from becoming an attack surface. An LLM that can only move a decision toward more human involvement cannot be talked into an approval by anything in the diff, which matters because the diff is attacker-controlled content in the general case (Agentic Prompt Injection).

Escalation is routing, not blocking#

A refusal doesn't dump the PR back into a queue; it routes to a subject-matter expert, selected by "CODEOWNERS-soft and git-blame familiarity." The gate's real product is therefore an attention-allocation decision, not an approval — it partitions the merge stream into "nobody needs to see this" and "this specific person should." That is the oversight-fatigue mitigation ("concentrate review on high-stakes decision points rather than every output") mechanized at the merge boundary, and it is why the throughput gain isn't purely subtractive: attention removed from 1.6K low-context stamps is available for the escalated remainder.

What it displaced matters for how you read it#

Before StampHog, PostHog ran a Slack channel (#dev-stamp-exchange, with a leaderboard) where an engineer dropped a PR and waited for someone to approve it. Yoon's own description of the cost: "each stamp required another engineer to take themselves out of their flow to approve a change they had little to no context on."

This reframes what the automation replaced. It was not substantive review being handed to a machine — it was an approval ritual that already carried near-zero information, performed by the least-informed available human. In the three-way acceptance partition from Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping?, the stamp exchange was already producing something close to bare non-reversion wearing a signature.

The honest reading, then, is that StampHog converts an implicit rubber stamp into an explicit, logged, gate-checked one. That is a real improvement in accounting — the auto-approved set is now enumerable, its criteria are written down, and the deny-list is a reviewable artifact — without being evidence that automated approval substitutes for review that was doing work. Reading "1 in 3 PRs approved by an agent" as "a third of review was automated away" over-claims; a large share of that third was ceremony.

Calibrate locally, or the deny-list is decoration#

PostHog's porting prompt refuses to hand over its thresholds: "Their deny-list and thresholds are calibrated to their codebase — re-derive for me: mine my git history for high-blast-radius deny candidates and calibrate size/tier ceilings from my merged PRs."

This is the load-bearing operational caveat. Both discriminating gates are empirical facts about one repository: which identifiers mark danger, and what "small" means in that codebase's merged-PR distribution. A deny-list copied across repos matches the wrong strings, and a 500-line ceiling imported into a repo whose median PR is 40 lines gates almost nothing. The transferable artifact is the derivation procedure (mine history for blast-radius candidates, fit ceilings to the merged-PR distribution), not the config.

And re-calibrate: the distribution the ceiling sits on is moving#

The porting prompt says to calibrate size ceilings from your merged PRs. It does not say to re-calibrate, and the ambient distribution is not holding still. DX's Q2 2026 panel (vendor-claim, 500+ organizations) reports median PR size nearly doubling between Q1 and Q2 2026, corroborating Faros's +51.3% on a calendar axis rather than an adoption cross-section.

A constant ceiling on a rising distribution loses coverage silently. Nobody edits the config, nothing errors, and the share of the merge stream that clears 500 lines and 20 files just shrinks — so "~1 in 3 PRs" is a measurement of one quarter, not a property of the design. Two consequences, pointing opposite ways:

• The decay is safe. Shrinking coverage under a fail-closed gate produces latency, not unreviewed merges. This is the invariant earning its keep against a threat its authors did not name.
• It makes the stacked-PR practice load-bearing rather than optional. The size gate only keeps paying if the authoring side is actively pushed down the size curve; otherwise the distribution walks out from under the ceiling and the gate quietly retires itself. Decomposition is what holds the auto-approvable band open.

The operational form is to express the ceiling as a percentile of the repo's own recent merged-PR distribution, recomputed, rather than as a constant — which also keeps it aligned with the 16.2%-to-53.6% smell gradient, since that gradient is a fact about absolute diff size and does not move when your median does. Caveat the transfer, though: DX measures a cross-industry panel median, not PostHog's repository, and no source measures whether any specific repo's distribution shifted.

The upstream complement: shrink the diff to fit the gate#

The source's fourth pattern turns the size gate into a design constraint rather than a filter. Daniel Visca's rule is observability over reasoning: "Agents are good at explaining why their code works. The explanation is often convincing... but also wrong" — so don't accept the argument when you can watch the behavior. Observation doesn't scale to a 3,000-line PR, so the agent is instructed to decompose the work into a stack of small single-purpose PRs (target <400 changed lines, stacked via Graphite), each independently runnable, each shipping "a command to run and the output I should expect," merged bottom-up so every layer builds only on already-observed behavior.

The two patterns compose, and the source says so: decomposition "lets StampHog from #3 auto-approve the small and focused PRs." The result is two checks of different kinds — the agent reasoning about the code, and a human observing it run — rather than two passes of the same kind. This is the merge-queue instance of the agent's explanation being the weakest available evidence, and the practical answer to the size gate's obvious objection ("most real work exceeds 500 lines"): change the unit of work, don't raise the ceiling.

Tension with the measured detection floor#

Security Debt of Agent-Generated Code (Sakib et al., arXiv 2607.12428, empirical) measured the surface this design gates on, and it cuts both ways. The empirical source outranks this case study wherever they meet — one is a coded study of 4,022 PRs, the other a first-party account with no efficacy measurement — so both readings below are the paper's, applied to PostHog's design.

It supports the size gate, strongly. Security-smell prevalence in agentic PRs climbs monotonically with change size, 16.2% for 1–9-line PRs to 53.6% for 1000+-line PRs. A 500-line/20-file ceiling keeps auto-approval inside the low-prevalence band, and the stacked-PR practice deliberately pushes work down that curve. That is independent, non-vendor, empirical backing for the one gate in the stack that looks most arbitrary — and it is the size-stratified evidence Faros asked for and could not produce.

It undercuts the deny-list, sharply. The deny-list is organized around business-risk topics (auth, secrets, billing, public APIs). The measured smell mass is somewhere else entirely: 82.3% is supply-chain integrity — mutable action tags, unpinned installs — and 87.6% of all smells live in GitHub Actions workflows and Dockerfiles, files that read as boilerplate and match no business-risk keyword. An unpinned actions/checkout@main in a 30-line workflow diff clears PR state, clears the deny-list, clears the size gate, and arrives at the LLM showstopper check as the only remaining defense — the layer whose measured performance on the most mechanically detectable smell class (hard-coded credentials, seven distinct commercial detectors present) was a comment on 18.9% of genuine live credentials. Build-and-deploy plumbing is precisely the blind spot of a deny-list built from application-risk intuitions, and it is where the debt actually accrues.

The concrete prescription this yields: .github/workflows/, Dockerfile, and IaC paths belong on the deny-list as paths, independent of keywords. PostHog's own instruction to mine git history for deny candidates would likely surface them; a copied deny-list of business-risk words never will.

The rival predicate: "the tests cover this change"#

The deny-list is a proxy for blast radius, and the obvious third proxy — the existing suite already exercises this diff — is the one gate in this space that is a fact about the change rather than a guess about it. Dipongkor et al. (arXiv 2607.18057, empirical) price it on 4,882 agentic PRs, and the price is low: existing tests execute 61.5% of agents' changed executable lines in Java and 27.0% in Python, with 64.8% of Python PRs having no changed line executed by any existing test at all. Their own prescription to practitioners is the negative form of it — "teams using agentic PRs should not assume a passing run of tests means that the change has been tested."

Two consequences for this gate stack, pointing opposite ways.

Against the stack as it stands: a green CI run is currently doing silent work in the "PR state" check and in every reviewer's prior, and for Python agentic PRs it is close to uninformative about the diff. Whatever confidence the four gates leave to "well, the tests passed" is, on this measurement, mostly unearned.

For extending the stack: unlike a keyword deny-list, diff coverage is computed from the diff. It needs no calibration to a codebase's vocabulary, it cannot be disarmed by a change that reads clean, and it produces a number rather than a judgment — which puts it squarely in the deterministic tier alongside the size ceiling rather than behind the LLM veto. The obvious form is a floor ("auto-approve only if every added executable line is executed by some test"), and it composes with the size gate rather than duplicating it: PR size predicts how much risk, coverage predicts whether anything watches it. The cost is the one the paper cannot estimate — a coverage floor may shrink the auto-approvable set much harder than a path deny-list would, since more than half of Python agentic PRs would fail it outright.

Note that this measures the same open-source agentic-PR population as the security study, has the same missing piece (no human-authored baseline), and does not stratify coverage by PR size — so it cannot say whether small diffs, the ones StampHog actually auto-approves, are better or worse covered than the corpus average.

Tiering on the anomaly instead of the diff: control bands (Anthropic, August 2026)#

Every gate above tiers on a property of the change. Anthropic's Applied AI AI-Native SDLC playbook (vendor-claim, 2026-08-21) proposes the same ladder keyed to a property of production, for work the agent originates rather than work a human handed it. A version-controlled bands.yaml sets response tiers against a rolling baseline on one metric with a stable distribution (CI test failure rate, post-deploy 5xx rate, PR cycle time):

• 1σ — log only.
• 2σ — invoke Claude read-only to diagnose, with the tools enumerated in the config (Read,Grep,Bash(gh run view *)).
• 3σ — Claude may act, but only through two pre-approved routes: open a PR into the existing review gate, or trigger a runbook approved in advance.

Three design choices are worth carrying across to a merge gate, and all three are this page's own principles restated:

• The detector is not a model. "Detection stays entirely deterministic, with no model involved" — a unit-tested, version-controlled script using rolling mean/σ with Western Electric-style rules to catch drift as well as spikes. The model is invoked by the gate; it is never the gate. Identical to demoting the LLM to a last-position veto here, and to the deterministic-first ordering at the tool boundary.
• The tier sets the action space, not the confidence threshold. Rising severity buys the agent more permitted routes, not more autonomy within a route; even at 3σ the output is a PR that a code owner still approves. This is the same one-directional authority as the veto — escalation is routing, not unblocking.
• Dismissals tune the bands. Triage outcomes feed back into the thresholds, which is the calibration discipline the deny-list here lacks.

What it shares with StampHog is the evidence gap, and worse: the playbook reports no deployment at all, so the tier boundaries have no observed false-positive rate, no noise budget, and no figure for how often a 2σ diagnosis is worth its token cost. σ-tiering also imports an assumption a deny-list does not — that the metric's baseline is stationary — and this page's own moving-distribution problem (a size ceiling decaying as ambient PR size rises) is the direct analogue: a rolling 30-day baseline absorbs a slow regression rather than firing on it, which is exactly the drift the Western Electric rules are added to catch and exactly where they are hardest to tune.

And the case study reports volume, not safety. 1 in 3 and 1.6K are throughput figures. P8/P9 is exactly this split — automated review reliably buys throughput and latency, its effect on quality and security is contested — and StampHog is a large deployed instance of the contested half with nothing measured attached to it. What would settle it is cheap and absent: the escaped-defect or incident rate of auto-approved PRs against the human-stamped baseline the channel used to produce.

The third tiering key: reversibility of the response (security operations, 2026-09-02)#

Two tiering keys are on this page already — a property of the change (StampHog's size ceiling and blast-radius deny-list) and a property of production (Anthropic's σ control bands). Security operations supplies a third, and the first population distribution of where teams actually set it.

Prophet Security's State of AI in the SOC 2026 (vendor-claim, vendor-commissioned survey, n=250 security leaders and practitioners fielded by ViB, every figure self-reported) asks AI-using SOCs what autonomy they grant a triage model. The answers sum to 100, so it is a single-choice ladder:

Highest autonomy granted	Share of AI users
Read-only triage	13%
Recommend actions, a human executes	44%
Auto-execute low-risk actions	30%
Auto-execute medium-risk actions	13%
Full unsupervised autonomy	0%

The tier is keyed to how hard the action would be to undo — quarantine, credential revocation, session termination, an access-control change — rather than to how large or how sensitive the input was. That is a different predicate from either of this page's, and it is the one that is computable from the tool call itself rather than from the artifact: an action's tier is a fixed property of the API you are about to hit, needing no calibration to a codebase's vocabulary and no baseline to drift against. It sits in the deterministic tier beside diff coverage, and unlike a keyword deny-list it cannot be disarmed by content that reads clean.

What transfers and what does not. The shape transfers exactly — deterministic tiering up front, the model bounded to a subset of permitted routes, escalation as routing rather than blocking. The level does not, and the reason is the same predicate: a merge auto-approval sits behind CI and a one-command revert, where an auto-executed containment action is customer-facing and often irreversible. Reading "0% full autonomy in the SOC" against "StampHog approves 1 in 3 merged PRs" as a difference in nerve gets it backwards; they are two settings of the same dial on actions with very different undo costs, which is exactly what a reversibility-keyed tier is for.

And the survey supplies one number this page has been missing for its own gate, in the wrong domain. Of AI users, 19% rely on vendor-reported accuracy metrics as their validation practice (against 57% requiring human review of every verdict before closure, 40% senior spot checks, 32% benchmarking against labeled datasets or red-team exercises, 5% no formal process — multi-select). This page's standing complaint is that StampHog reports volume and never efficacy; in the SOC, roughly a fifth of teams close that same gap by accepting the seller's accuracy figure in place of their own measurement. That is not a control, it is a vendor-claim standing where a measurement should be, and it is the outcome the "no escaped-defect rate" question below drifts toward if nobody runs the before/after.

The fourth tiering key: consequence of the output (Anthropic × Accenture, 2026-09-11)#

Three keys are on this page — a property of the change, a property of production, and the reversibility of the action. [[raw/claude-accenture-pilot-to-production|Deploying AI from pilot to production]] (Anthropic × Accenture, vendor-claim) supplies a fourth, keyed to what the output does when it lands, and it is the first one on this page written for non-code enterprise workflows rather than for a merge queue or a SOC.

Tier	Oversight model	Example use cases	Review cadence
1. Automated	No human review; output goes directly to workflow	Summarizing meeting transcripts; reformatting data between systems; generating code documentation	Quarterly audit of output quality
2. Sampled	Random subset reviewed on a regular cadence	Extracting structured data from invoices; classifying support tickets; populating CRM fields from call transcripts	A portion reviewed weekly; start higher and adjust as error patterns stabilize
3. Reviewed	Human approves every output before it reaches its audience	Drafting client communications; financial analysis for external distribution; contract language from templates; marketing content	Every output reviewed before release; audit the review process monthly for catch rate and cost
4. Advisory	AI provides analysis, a human makes the decision and produces the output	Credit and lending recommendations; candidate screening; clinical findings for physician review; regulatory filing preparation	Every decision documented with an audit trail; compliance review quarterly

The predicate is neither the artifact's size nor the action's undo cost but who the output reaches and what they do with it — which is why "generating code documentation" sits at tier 1 and "drafting client communications" at tier 3 despite being the same act of text generation. Tier 4 is the one worth noting: it is not a stricter review, it is a different architecture — the human produces the output, so there is no AI output to review at all, and the gate has been moved upstream of the artifact rather than placed in front of it.

The cadence column is the part this page did not have. Every earlier key on this page specifies which items get reviewed and none specifies how often the tiering itself gets re-examined. Here each tier carries its own audit rhythm, and tiers are explicitly declared mobile: "if a process has had a low error rate for a long period of time, say six months or a year, it may be ready to move up to tier 2." Tiers as a ratchet that only tightens is the failure the next section names.

Structural drag, and the checkpoint audit#

The document's account of why oversight accumulates is the sharpest thing in it:

"review checkpoints accumulate because they're easier to add than to remove, and because the risk of removing a checkpoint is visible while the cost of keeping it is not."

The asymmetry is between a legible risk and an illegible cost, which is why the drift runs one way regardless of anyone's intent. The prescribed counter is a three-question audit run periodically against each manual review step:

1. What does it actually catch?
2. How often does a reviewer change the output?
3. What is the cost per review, and what justifies incurring it?

Low catch rate plus high cost means the checkpoint gets automated, sampled, or removed. Question 2 is the load-bearing one and is the same measurement Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? argues is what separates a real control from a ritual — a reviewer who never changes the output is not reviewing, and this is the first source in the corpus to write that as a standing operational metric rather than as a critique. It is also, in this document's own framing, the mechanism by which StampHog's predecessor was diagnosed: an approval ritual whose catch rate nobody had ever measured.

Tier note. All of this is prescriptive vendor guidance with no deployment behind it — the document reports no organization running the ladder, no tier distribution, and no catch rates from the audit it prescribes. Read it as a well-argued taxonomy, not as evidence. Its companion claim from the same source — that organizations with formal chargeback accountability tie 32¢ of every AI-token dollar to a quantified business outcome, six times those with no allocation — is the closest thing it offers to a measured governance outcome, and it measures cost attribution rather than oversight efficacy.

Connections#

• Autonomous Defense — the same ladder in security operations, keyed to the reversibility of the response (see the section above): 13% read-only / 44% recommend-only / 30% auto-execute low-risk / 13% medium-risk / 0% full autonomy, self-reported by AI-using SOCs in a vendor-commissioned survey of 250 security leaders. It is the first population distribution of a risk-tiered gate anywhere in the corpus — this page has one deployment and no distribution — and it arrives by policy rather than by a gate stack, which is why the tier is stated in terms of the action rather than the artifact

• Misalignment in Production Agent Traffic — the informed-consent boundary this page's gates depend on, written as an explicit judging criterion. A named bypass ("use --no-verify when you commit") or a short approval given after the agent surfaced the gate both authorize; "commit this", "just get it done", "don't ask me questions" and general grants of autonomy do not — "a reflexive approval from a user who was never told about the gate or the bypass is not informed intent, however broad it sounds." Project documentation instructing a specific bypass counts; the agent's own inference from ambiguous docs does not

• Same-Model Review Blindness — a free, unclaimed tightening of the last gate. Three of the four checks are deterministic and the fourth is an LLM showstopper running last on a diff a coding agent produced; nothing in the design constrains which model runs it. Greptile's paired datasets (case-study) put 6–12 points of high-severity recall on that choice — each frontier model catches fewer bugs in its own family's code — so routing the veto to a different family than the authoring agent is worth more than any prompt tuning, costs nothing, and needs no calibration to the codebase the way the deny-list does. It is also the cheapest possible instance of the independence spec this same source states for its reviewer panel ("different models and providers for different reviewers"), applied to the gate rather than the panel. Note the ordering caveat: the veto sits behind three gates by design, so this raises the weakest layer without changing what reaches it

• Agent Review Comment Resolution — independent empirical support for two of this design's least-argued choices. Across 54,713 agent review comments, an inline code suggestion is the strongest predictor of resolution (OR 1.62, 75.5% vs 64.6%) while comment length carries a penalty concentrated in functional feedback (OR 0.855) — which is StampHog's minimal-output rule (a bare approval with no line comments; a refusal of 1-2 sentences plus a risk level and next steps) measured rather than asserted. And it validates demoting the model to a veto: a veto produces no comment, so it has no resolution rate to lose, and it sidesteps the failure that dominates that study's rejections — an agent flagging as a defect what the team decided on purpose (23.8% of argued cases). What it does not supply is the escaped-defect number this page's first open question asks for; adoption is not correctness there either

• Verification as the New Bottleneck — a partial, deployed answer to that page's "how far do you push fully automated reviews?": as far as the cheap, structural checks reach, with the model demoted to a veto behind them — not as far as a model's judgment about correctness. That page's CircleCI material (vendor-claim) is the same cheap-checks-first ordering pushed one stage earlier: run lint, unit tests, and builds in the inner loop so failures surface in seconds, and CircleCI models ~$700K/yr of a 50-developer team's delivery cost as recoverable that way. Both are ordering arguments — put the checks that are fast and certain ahead of the ones that are slow or probabilistic — differing only in where the cheap tier sits (the developer's loop vs the merge gate), and the composition is the obvious one: a diff that already passed the inner loop arrives at StampHog cleaner

• Review as the Control Point — the deployed instance of P17 (risk-tiered gating of material changes only), and a large uncontrolled instance of the contested P9: this reports throughput (1 in 3, 1.6K/month) and measures no quality outcome

• Security Debt of Agent-Generated Code — the empirical check on the gate stack, cutting both ways: the 16.2%→53.6% size gradient backs the diff-size ceiling, while 87.6% of smells living in CI/container files that match no business-risk keyword exposes the deny-list's blind spot, and the 18.9% credential-comment rate is the measured floor under the LLM showstopper check that would have to cover it

• Agent-Generated Test Quality — the rival gate predicate, measured (see the section above): existing tests execute 61.5% of agents' changed lines in Java and 27.0% in Python, and reach nothing at all in 64.8% of Python PRs — so "the tests passed" is not the reassurance the gate stack implicitly leans on. It is also the one candidate predicate that is computable from the diff, which makes it eligible for the deterministic tier where a keyword deny-list is a guess and the LLM check is a judgment

• Deterministic Pre-Execution Gates — the same deterministic-first ordering at the tool-call boundary, and the complement to this page's evidence gap. Two differences decide how they compose. StampHog's gates are proxies — a keyword deny-list and a line count stand in for "would a mistake here be expensive" — where a policy gate decides the actual question it was written for, which is why gate precision there is auditable (100% over 161 fires for one gate, 5% for another) and deny-list coverage here is not. And the reporting is mirrored: this case study measures volume and never efficacy, that paper measures efficacy (+12.4pp, replicated on disjoint seeds) and has no deployment. The transferable rule both support is the ordering, not either config

• Claude Code Auto Mode — the same classifier-gated-autonomy shape one layer down (per-tool-call rather than per-merge), with the same one-directional authority: the model may block, never unblock. The difference is what carries the decision — auto-mode's classifier is the gate, where here three deterministic checks gate and the model only vetoes

• The Committed-Artifact Chain — the process this gate would sit inside, and the source of the control-band variant above. The playbook needs tiering at three gates (accept the spec, accept the plan, merge the PR) and specifies it at none of them — it says only that "anything the organization classes as higher risk" routes to a tech lead — so StampHog is the calibrated instance of a lever that playbook leaves as an exercise. The chain also supplies a predicate this page does not have: a committed plan.md makes "the diff still matches what was approved" computable, which is a fact about the change in the same class as diff coverage

• Acceleration Whiplash — the lever against the review-time explosion (median time-in-PR-review +441.5%), and the mechanism by which review coverage can rise while human review-hours fall; but Faros's thesis is that the fix belongs in authoring, and a merge gate is downstream of that

• AI Brain Fry — the interruption side: what the gate removes is 1.6K context switches by engineers approving changes "they had little to no context on," and escalation-with-routing is the "concentrate review on high-stakes decision points" mitigation mechanized

• Optimizer–Evaluator Decoupling — the same source's reviewer-panel pattern applies the rule with an explicit independence spec: the agent that wrote the code can't review it, and reviewers should span different instructions, models, and providers

• Loop Engineering — the auto-stamper and the PR-babysitting loop are two loop-engineering products aimed at review-adjacent toil; PostHog's Paul D'Ambra puts a price on the discipline ("something like 60% of my token spend is burned automating the toil of handling CI and review")

• Blast Radius (Agentic) — a third sense of the term in the vault: here it is neither security-compromise scope nor code-change footprint but a merge-gate predicate — a keyword deny-list standing in for "would a mistake here be expensive," which is why the substitution's coverage is checkable and, per the security study, incomplete

• Unproductive Self-Verification — "verify by observation, not reasoning" is the reviewer-side statement of why an agent's account of its own work is the weakest available evidence; decomposing until each diff is runnable is the escape

• Polish No Longer Signals Readiness — what the deterministic gates route around: a size and path check cannot be disarmed by a diff that reads clean, which is the specific reviewer failure (P4) that a model-first gate inherits

• Write-Then-Trusted — the deny-list critique's second instance, one layer down: Pillar Security's verdict on allow-by-default sandbox profiles — "not a sandbox, it's a list of things somebody remembered to block that's always one entry short" — is the OS-policy version of this page's measured gap, where a blast-radius keyword deny-list misses the CI/container files carrying 87.6% of measured security debt. Two independent enumerate-the-bad controls, each falling short in the direction its author could not anticipate; the open question below ("does extending the deny-list restore coverage?") has the same shape in both

• Deterministic Engineering for Agent Code Review — the same deterministic-before-model ordering at a smaller join. OpenCodeReview's comment-anchoring step tries two cheap deterministic matches (the cited snippet against the diff's new-side hunks, then against the full file) before ever falling back to an LLM relocation call — the same model-only-after-the-cheap-checks-fail shape this page's gate stack and showstopper veto share, applied to where a comment attaches rather than whether a PR merges

• Pilot-to-Production Gap — where this page's tiering sits in an enterprise deployment lifecycle. Consideration 06 of that blueprint is the fourth tiering key above, and the framing it adds is that oversight design is an in-production decision whose owner ("risk and monitoring owner") must be named pre-pilot — so the gate's calibration is an org-chart question before it is an engineering one. It also supplies the structural-drag account of why every gate on this page drifts toward more review than it can justify

• AI-Assisted Error Analysis — the same consequence-tiering logic applied to eval investment rather than merge autonomy: a uniform accuracy bar across every AI feature is the mistake, and the bar gets set by reasoning through worst-case user outcomes up front (with an assistant enumerating them for a human to judge), then working backwards into which evals and guardrails to build

Derived#

• Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — the oversight synthesis this bears on: a deployed calibration of its Answer 3 partition, and a case where the automated layer replaced an approval ritual that was already rubber-stamping — so it changes the accounting of the merge stream more than it changes the safety of it

Open Questions#

• The case study reports volume and never efficacy. What is the escaped-defect or incident rate of auto-approved PRs versus the human-stamp baseline the Slack channel used to produce? PostHog has both populations in its own history, which makes this a checkable before/after rather than a request for new instrumentation. The same history answers a second question for free (added 2026-08-12): plotting the share of merged PRs that would clear the 500-line/20-file ceiling, quarter by quarter, dates how fast the gate's coverage decays as the ambient PR-size distribution rises.

• A keyword deny-list is a proxy for blast radius, and Security Debt of Agent-Generated Code shows the proxy misses where the measured debt concentrates (CI/container plumbing, 87.6%). Does adding CI/IaC paths to the deny-list restore coverage, or does it shrink the auto-approvable set so far that the gate stops paying for itself? A third candidate predicate is now priced but not tested (2026-08-12): Dipongkor et al. (empirical) show that "the existing suite covers this diff" is computable, deterministic, and rarely true — 27.0% of changed lines in Python, nothing at all in 64.8% of PRs. That reframes this bullet as a choice between two extensions with the same failure mode rather than one open question: both a path deny-list and a coverage floor buy precision by shrinking the auto-approvable set, and neither has been measured against the volume it costs. The same before/after PostHog already has in its history would answer both at once.

• If the size ceiling becomes a design target — agents instructed to emit stacks under 400 lines — does total risk fall, or does it just redistribute into more PRs each below the gate, with the integration risk moving to the seams between them? The security study's size gradient is measured per-PR and cannot distinguish these.

• Three tiering keys are now on this page — the change (size ceiling, deny-list), production (σ control bands), and the reversibility of the action (the SOC ladder). Only the first has a deployment with a coverage number attached. Does a reversibility-keyed tier admit a larger auto-approved set at equal escaped-defect rate than a size-keyed one? Checkable in the same PostHog history the first question asks for: partition merged PRs by how cheaply the change could be undone (revert-only versus data migration, config, or a released artifact) instead of by line count, and compare the auto-approvable share and the incident attribution of each partition. The prediction from the SOC ladder is that reversibility dominates size, since every gate that tiers on undo cost stops at exactly the point where undo stops being possible.

Sources#

• The State of AI Impact in Engineering: Q2 2026 — Justin Reock, The State of AI Impact in Engineering: Q2 2026 (DX, Engineering Enablement newsletter, 2026-07-22), tier corrected empirical to vendor-claim at compile (reasoning in Sources). Finding 2 only — median PR size nearly doubling Q1 to Q2 2026 across 500+ customer organizations — cited here as the moving distribution under this gate's fixed ceiling. No methodology accompanies the figure; the report is gated

• Stop being the code review bottleneck — Jina Yoon, "Stop being the code review bottleneck", PostHog newsletter (2026-07-09), case-study. §3 (StampHog: label trigger, four gates, invariants from the porting prompt, escalation routing, 1-in-3 and 1.6K figures, the #dev-stamp-exchange predecessor), §4 (observability over reasoning; stacked PRs under 400 lines via Graphite; the explicit composition with §3), §1 (reviewer panel, independence across models and providers, the 60%-token-spend quote)

• State of AI in the SOC 2026: 8 Key Takeaways — Ajmal Kohgadai (Prophet Security), State of AI in the SOC 2026: 8 Key Takeaways, 2026-08-03, vendor-claim (vendor-commissioned survey, n=250, fielded by ViB, self-reported, methodology behind a lead-gen form; Prophet sells an agentic AI SOC platform). Cited here only for §7 — the single-choice autonomy ladder and the validation-practice multi-select. Full treatment on Autonomous Defense; tier reasoning in Sources

• Deploying AI from pilot to production: A practical blueprint for CIOs and technical leaders — Deploying AI from pilot to production, Anthropic × Accenture, 2026-09-11, 38pp, vendor-claim. Consideration 06 only — the four-tier oversight ladder with its review cadences, the structural-drag account, and the three-question checkpoint audit. Prescriptive throughout: no deployment, no tier distribution, no catch rates. Full treatment on Pilot-to-Production Gap