Critique Triage

Sources#

• Claude Engineering Lead: "Stop Using Claude Like It's 2024"
• Helix: The internal tool powering our Shopify app's native migration
• Rewriting Bun in Rust
• Stop being the code review bottleneck

Summary#

Adversarial review gives each reviewer a job: prove the output wrong. Reviewers with that job produce criticism whether or not the output deserves it, so a pipeline that runs them has to decide what to do with the criticism. This page is about that step. Optimizer–Evaluator Decoupling covers who reviews and what they may see. Same-Model Review Blindness covers which model reviews.

Felix Rieseberg, who leads engineering for Claude.ai, Cowork and Claude Code Desktop at Anthropic, gives the most explicit version in the corpus (Claude Engineering Lead: "Stop Using Claude Like It's 2024", Silicon Valley Girl interview, 2026-10-20, practitioner-opinion, first-party). He gives it as his answer to "how do I protect myself from AI confidently presenting mistakes":

1. Attack. "You've given one Claude a task, spin off another 10 that try to like break this down. Their whole task is to attack this and prove to you why it's wrong. It's essentially the same concept as red teaming or short selling." For a blog post he wrote that morning, he told Claude to "fire off five threads and task each of them with like proving that this post is wrong, that I'm thinking about it the wrong way, that I have my facts wrong, that I misunderstood something."
2. Judge. "What we use in coding context is something we call a judge": a further session that takes the attackers' main criticisms, and whose job is "to validate and verify and like opinionate whether or not this is valid feedback."
3. Author's discretion. The valid feedback goes "back to the Claude that actually made the thing", and that agent "too can be discerning [about] whether or not to act on feedback," because "sometimes the feedback is like maybe reasonable but has missed a point or it's actually not that actionable or is not that useful."

The third step also gives a stopping signal: when the surviving criticism is reasonable but unactionable, "that's usually when I know that my output is pretty good."

The aggregation rules in the corpus#

Four deployed loops, and four different answers to what happens between "a reviewer said X" and "the code changes":

The rows split on one question: is a reviewer's finding a verdict or evidence? Helix treats it as a verdict. The pipeline cannot override a failed check, and the stricter verdict wins. That fits a gate whose standard is written down (Shopify's architecture docs), so a finding is checkable against a document. PostHog and Rieseberg treat findings as evidence that has to be weighed. That fits review against an unwritten standard, such as a blog post's argument or a PR's rollout risk, where an attacker told to find fault will produce some faults that are not real.

The human-side base rate says the filter is needed. When agents review and humans decide, about seven in ten agent review comments are resolved (Agent Review Comment Resolution). The modal reason the rest are not is project context the reviewing agent could not see (23.8% of the coded discussions), followed by confident false positives. In those repositories the human resolving comments is the judge. Rieseberg's loop moves that job to another model session.

Two cost claims#

• Reviewers can be a cheaper model. "It is sometimes easier to find flaws than to create new things… So usually you can like take a step down in intelligence and still get very good results." This matches Cursor's cost argument on Optimizer–Evaluator Decoupling ("review is much cheaper than the work it audits"), approached from the capability side rather than the token side. Neither is measured.
• Same model or a different one: "kind of depends on the task." The empirical evidence is more specific. Greptile's labelled PR sets (empirical) find each frontier model catching fewer bugs in its own family's code (Opus 4.7 53.7% same-family vs 60.0% cross-family). For bug-finding, that measurement outweighs an unmeasured "it depends." Rieseberg's own example, Claude attacking a Claude-assisted blog post, is the same-family case.

When to run it, and when not#

Rieseberg sorts outputs into two buckets. If you "really have no idea what good looks like" and cannot check the output yourself, "my answer is more AI, just throw more tokens at it": run the attack-judge loop. If you can judge the result as a client would, that may be enough. His example is a personal website rebuilt with 3D assets in Blender, which he has never used. He judges whether it "looks good, feels good" and has no opinion on the process. This is the contractor, lawyer or doctor relationship. It is the same split Verification as the New Bottleneck draws between outputs whose result you can inspect and outputs whose correctness lives in the process.

His one fully trusted single-agent task is the opposite of self-judging: editorial fact-checking against primary sources ("I need you to find primary sources for every number I mention… I fully trust that 100%"). It is consistent with the loop above, because a primary source is the "hard outside evidence" he prefers to another session's opinion. The 100% is a self-report. No accuracy figure is given.

Why not the session that made it#

"I rarely want the current session to judge itself. That I think is still really iffy… the current session tends to be quite bad at being a judge of whether or not the work it did was quite good." His order of preference is hard outside evidence first, another session "in a pinch." That is Optimizer–Evaluator Decoupling's invariant stated by a vendor engineering lead for chat, projects and Cowork, not just for coding. It also gives a reason to delete self-verification instructions (Unproductive Self-Verification): a check by the author session is weak evidence, so paying for it twice buys little.

Trust built side by side#

For the bottleneck of a manager who reviews every output, Rieseberg's recommendation is a shadow run: keep writing your own feedback, record what Claude would have said for "a few rounds," and compare where they differ. He grants that humans themselves disagree run to run ("the same human… on a Monday versus a Friday"). Alongside it he describes a three-step feedback distillation: assemble all the feedback you have given, have Claude condense it into rules or principles, optionally run a third pass for "principles that maybe you might believe but you don't actually know you believe," and feed that list into later reviews. It "has been quite successful in our attempts to… specialize the feedback that individual sessions give." Both are judge calibration done informally, with the human as gold standard. LLM-Judge Validation is the measured version, where the agreement statistic is reported rather than eyeballed.

Limits#

• Every row in the table is self-reported. No source compares judge-filtered against union-of-findings aggregation on the same outputs, and none reports a judge's false-reject rate, meaning valid criticism it threw away.
• The judge is another model session, so the loop has two noisy verifiers in series. Stopping Under a Noisy Verifier shows that verify-repair loops against noisy verifiers can raise reported acceptance while true quality falls. Rieseberg's stopping signal is a heuristic, not a boundary computed from verifier and repairer error rates.
• The author's veto reintroduces the self-judging he distrusts. The session that made the output decides which surviving criticism to act on. The judge stage limits the damage, because the author only declines feedback that has already passed an independent ruling, but the final call is the author's.

Connections#

• Optimizer–Evaluator Decoupling — the invariant this sits inside: whoever made it does not grade it. That page carries the Bun spec (context asymmetry, inverted prior) and PostHog's panel. This page carries what happens to the findings afterwards
• Checkpoint-Gated Convergence — the no-triage pole: union of findings and stricter verdict wins, enforceable because the standard is written architecture documentation
• Dynamic Workflows: An Algebra for Agents — Bun's implementer / adversarial reviewers / fixer loop, which names the fixer and leaves acceptance unstated
• Same-Model Review Blindness — the measured answer to Rieseberg's "same model or another? depends": for bug-finding, the reviewer's family matters
• Agent Review Comment Resolution — the human-as-judge base rate: about 71% of agent review comments resolved, with missing project context and confident false positives as the leading reasons the rest are not
• Unproductive Self-Verification — why the author session is the wrong judge, and why paying it to re-check adds little
• Stopping Under a Noisy Verifier — the formal stopping rule Rieseberg's "reasonable but unactionable" heuristic approximates without the error rates
• LLM-Judge Validation — the measured form of the side-by-side shadow run: score the judge against the human before trusting it
• Generalist Session over Agent Cast — the same source's position that the reviewers are sessions given a job, not personas given an identity
• Verification as the New Bottleneck — the two-bucket rule: throw tokens at outputs you cannot check, use your own judgment on outputs you can
• Felix Rieseberg — the source of the attack-judge-author spec
• Cowork — the knowledge-work surface he applies it on

Open Questions#

• Does a judge stage between adversarial reviewers and the author improve review precision without losing recall, compared with acting on the union of findings? Settler: an ablation on a labelled review set (such as Greptile's 500-PR datasets) scoring union-of-findings, triage-classifier and judge-filtered aggregation for true and false findings acted on.

Sources#

• Claude Engineering Lead: "Stop Using Claude Like It's 2024" — Felix Rieseberg interviewed by Marina Mogilko, Silicon Valley Girl, 2026-10-20, ~49 min, practitioner-opinion, first-party (he leads engineering for the products he describes). Chapters "How to catch AI's confident mistakes" (the attack/judge/author loop, step-down reviewers, the two buckets, fact-checking), "Why 'you're an expert' isn't enough" (self-judging, outside evidence, feedback distillation) and "How to stop being your company's bottleneck" (the side-by-side shadow run). The transcript is auto-captions with a scripted ASR fix list and inferred speaker labels. None of the quotes used here falls in the turns the raw marks as uncertain attributions
• Stop being the code review bottleneck — Jina Yoon, PostHog, 2026-07-09, case-study: Paul D'Ambra's qa-swarm (four reviewers) and review-triage (actionable / nit / ambiguous, up to three rounds)
• Helix: The internal tool powering our Shopify app's native migration — Talha Naqvi, Shopify Engineering, 2026-09-21, case-study: Gate 3's "union of findings, stricter verdict wins", from the gate diagram
• Rewriting Bun in Rust — Jarred Sumner, bun.com, 2026-07-08, case-study: the implementer / two adversarial reviewers / one fixer loop ("1 fixer applies the fixes")