Code Review Agents on Human-Verified Bugs (CRBugBench)

Sources#

• (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

Summary#

Every earlier review-efficacy number in this vault measures something next to defect detection: adoption of comments (Agent Review Comment Resolution), semantic match against all human comments, nits included (Deterministic Engineering for Agent Code Review), or recall against a vendor's own labels (Same-Model Review Blindness). Yang, He, Cheng, Butler, Murphy-Hill, Badea and Bird (Carnegie Mellon and Microsoft / Microsoft Research, arXiv 2610.13668, 2026-10-09) build the instrument that targets detection directly. The ground truth is bugs a human confirmed by fixing them. Each bug is traced back to the PR that introduced it, the agent reviews that PR in a sandbox, and the score is whether the bug appears in its top-k findings.

The question in the title is whether a "lights-off" software factory, where AI writes and reviews code with almost no human supervision (the term is Dan Shapiro's "dark factory"), is ready. The answer is not yet, and not uniformly. The best configuration catches 71.9% of known bugs in its top five findings, and the misses concentrate where nobody can see them from the diff alone: large changes, mixed changes, bug-fix PRs, non-local bugs, and proprietary monorepos. The authors' verdict is that the evidence "supports 'lights-off' operation only for narrowly scoped, low-consequence PRs". For consequential changes, a review agent should be a scalable peer and an accountable human should keep the final decision. That is the Risk-Tiered Auto-Approval gate in benchmark form.

Evidence note. empirical, tier kept. A large measured benchmark with stated validation: 37 of 40 sampled tasks valid (92.5%, Wilson 95% CI 80.1–97.4%); a three-family judge panel with human–LLM κ = 0.89 on 40 samples and Fleiss κ = 0.95 across judges; regression with task-clustered errors and Benjamini–Hochberg correction. It is an arXiv preprint, not yet peer-reviewed, and the full replication package is withheld (an open-source subset is on GitHub). COI, two-sided: all authors but the first are Microsoft employees and the first did the work at Microsoft Research. The primary harness is GitHub's copilot-cli (Microsoft), and the one harness comparison favours it over Claude Code. The model rankings, by contrast, put Anthropic models on top. The proprietary repositories are at "one large company", unnamed, and are almost certainly the authors' employer's.

The instrument#

Tasks come from two mining pipelines over 106 repositories (five languages, 31 application domains, collected 2026-06-26):

• Cmt. track: bugs caught before merge. From 1,436,120 review threads, a deterministic filter and then an LLM filter remove style, naming, nit and declined comments, leaving 865 vetted bugs. These are review comments that were addressed.
• Fix. track: bugs caught after merge. From 195,075 fix-keyword PRs to 57,725 with substantive linked issues, then traced to the inducing PR with SZZ, then LLM-screened for refactor misattribution and for poorly described issues, leaving 689 pairs. These are defects that shipped past whatever review the PR got.

Each track is sampled to 500 buildable Docker tasks. The open-source benchmark is 1,000 tasks over 98 repositories (median churn 189 lines, 82% single-language). The same pipeline on two industry monorepos (average >100k source files, against ~3k for the open-source set) adds 800 proprietary tasks. The agent gets the repository at the base commit with all later history deleted, the PR title and description, and the diff. It is not told a bug exists, and it writes a severity-ranked review.md with no Internet access. A three-judge majority (gpt-5.5, claude-opus-4.8, gemini-3.1-flash) marks each finding as a match or not. The panel is cross-family on purpose, to blunt same-family judge bias (LLM-Judge Validation). The metric is recall@k.

What the construct leaves out. It scores one known bug per task and never penalises a false positive. Precision, noise, and any useful comment that is not the planted bug are invisible to it. The authors say so themselves: the results are "recovery of known bugs rather than overall review usefulness". This is the mirror image of AACR-Bench, which penalises every comment that matches no human comment. Neither is wrong, but they do not average (see Reconciling below).

RQ1: how well, and how fast it is moving#

All runs use copilot-cli 1.0.74 with its stock /review prompt at xhigh effort or the maximum available. Numbers below are reconciled against the PDF's Table 1 (see the parse warning in Sources). Recall@5 is given as Cmt / Fix, and cost per task is the mean of the two tracks.

The headline numbers reproduce from the table. Opus 5 averages (79.6 + 64.2) / 2 = 71.9 and $2.97. Opus 4.6 averages 47.7. gpt-5.6-luna reaches 78% of Opus 5's recall at 21% of its cost. Pre-merge bugs are much easier than post-merge ones for every model, by 15–20 points. That fits: the Fix track holds bugs that already got past a human.

The trend is mostly breadth, not a better top pick. This is our reading; the paper does not say it. From Opus 4.6 to Opus 5, Cmt recall@1 rises only 45.4 → 48.3, while recall@5 rises 59.1 → 79.6. Opus 5 is barely ahead of Sonnet 5 at @1 (48.3 vs 47.8). It also emits about twice as many findings as any other model (5.76–6.78 against 2–3.6). For every other model, recall@5 ≈ recall@10 ≈ recall over the whole review, because the lists are shorter than five. So much of the 47.7 → 71.9 "capability" gain is Opus 5 saying more, ranked well. Fig. 5a shows 89% of eventual hits sit in the top three. That is real value at k = 5. But a recall-only metric cannot price the extra findings, and the paper's own default prompt asks for signal-to-noise. The Opus 5 prompting guide reports longer output by default, and this is consistent with it.

Humans and agents, by construction. Pre- and post-merge bugs are nearly equal in the mined population (49.7% / 50.3%). So "the human reviewer" catches 50% of the union by definition, the best agent catches 71.9%, and the union is 82.1% (Fig. 5b). This is a counterfactual overlay, not a measured collaboration. The interaction model also shows the two catch different things. Humans are much better on maintainability bugs (OR 2.702) and worse on severe (OR 0.450) and reliability (OR 0.591) bugs.

Proprietary code. On the 800 industry tasks, every evaluated model drops. Fig. 2a reads about 72 → 53 for Opus 5 and about 57 → 37 for gpt-5.6-sol; the text says "up to 25.5%". Only 22.8% of proprietary bugs are visible in the diff, against 46.1% on GitHub, and adjusting for visibility alone explains 20% of the gap. The rest is unattributed: scale, domain knowledge missing from training data, and possibly familiarity with the public repositories, which the paper does not test (see Open Questions).

Model upgrades are not monotone. Pairwise migration between models trades gains for regressions of up to 8.9% of tasks (Fig. 5c). Opus 5 is the exception, regressing on at most 1.2%. Its edge over the others is concentrated on feature and bug-fix PRs and narrows significantly on refactors (ratio of odds ratios 0.568) and on other PR types (0.476).

RQ2: where detection fails#

A multivariable logistic regression with task-clustered standard errors and BH-corrected tests found:

• Change complexity hurts. Log changed lines OR 0.591 (**), and diff heterogeneity (several change types, extension entropy) OR 0.795 (). Dependency fan-in and fan-out, repository size and language are not significant once diff size is in the model.
• Bug-fixing PRs are harder (OR 0.521, ***). The authors' explanation, offered without a test, is that attention goes to checking the fix and not to the regression it introduces.
• The bug's character matters most. Diff-visible bugs OR 2.621 (), severe bugs OR 1.906 (), usability bugs OR 0.454 (**), portability OR 0.523 (p = 0.07). Agents are tuned toward high-impact functional defects, so less severe and non-functional ones get neglected.

These are the same predictors the human code-review literature found (change size, composite changes, delocalised defects). It suggests a review agent's blind spots are mostly the task's blind spots, not exotic machine failures. The practical consequence is that change size is a usable tiering key for automated review as well as for human review. That is independent support for the size ceiling in Risk-Tiered Auto-Approval.

RQ3: what agents do, and how they fail#

• They read; they rarely run code. Runs open with git (64%, mostly git diff/git log), and then reading (39%) and searching (36%) take 75% of turns. Execution takes 8%. Claude models execute more (Opus 5 9.7% vs gpt-5.6-sol 6.9%), and two of Opus 5's unique catches came from running things: it pulled ~25 Wayback pages to exercise a yt-dlp extractor, and it built an MSBuild probe to prove a DLL was missing from the publish output.
• They stay for different lengths. Half of gpt-5.6-sol runs end by turn 17 and 3.5% reach turn 50. Opus 5's half-life is turn 35. That is a budget difference the harness never set.
• Failures are diagnosis, not retrieval. 95.6% of runs open the defective file, 80.2% write a finding on it, and 56.2% name the bug. For Opus 5, 91.9% write a finding on the file and about 75% of those diagnose it correctly. "Better search" is not the missing piece.
• They overtrust what the author says. In a Selenium task every model endorsed a buggy condition, because both the code comment and the PR description asserted the intended behaviour. In an Elasticsearch task, Opus 5 used the defective line as its reference for correct behaviour. The authors connect this to context-grounded hijacking of agentic workflows: a reviewer that takes the PR description as evidence can be steered by whoever writes it. This is the review-side version of Failures That Look Like Success.

RQ4: what buys recall#

Measured on average recall@5 across the two tracks, from Table 2 (reconciled) and Table 3 (clean):

• Effort is the steepest lever. gpt-5.6-luna goes from ~36% to ~57% between low and max effort, at 23.22× the cost.
• A "comprehensive" prompt is the cheapest lever. It asks for every finding and then a severity ranking, replacing the stock signal-to-noise instruction. Results: Opus 4.8 +5.75 at 1.10× cost (the abstract's "up to 5.8%"), gpt-5.6-sol +5.3 at 0.90×, gpt-5.5 +4.7 at 0.90×. Opus 5 gets −0.75 (+0.3 Cmt, −1.8 Fix) at 1.07×. The authors' reading is that "many of the encouraged behaviors are gradually incorporated into model training", which is the Harness Shrinkage as Models Improve thesis measured on a third party's harness. Combined with the breadth observation above, there is a sharper reason: Opus 5 was already emitting the long list the prompt asks for. This is also the first third-party ablation of the "report everything, filter in a separate pass" rule from Anthropic's Opus 5 prompting guide (see Writer/Reviewer vs Agent-to-Agent Review and Review as the Control Point). The rule pays for every model that follows a conservative prompt literally, and is moot for the one it was written for.
• Multi-agent orchestration is the expensive lever. For gpt-5.6-sol, the five orchestrations average +2.8 at 3.68× cost. The best is a five-candidate ensemble at +4.65 for 5.54×. Sub-agents lose 3.7 points on the pre-merge track while gaining 6.0 on post-merge. On the Pareto front (Fig. 2b), comprehensive-prompt configurations dominate every multi-agent one. See Multi-Agent Collective Intelligence for the general case.
• Mixing model families did not help the ensemble. A three-candidate ensemble drawn from claude-opus-4.8, gpt-5.5 and gpt-5.6-sol scores 66.8 / 52.8. Three candidates from gpt-5.6-sol alone score 68.0 / 53.6, at similar cost ($5.57 / $6.91 against $5.14 / $7.48). The differences are within a point and untested. Still, the point estimates give no support to the idea that decorrelated reviewers stack (Weak-Verifier Ensembling). This is not evidence against Same-Model Review Blindness. That result is about the reviewer's lineage relative to the author's, and every author here is human. Reconciled with a controlled result that looks opposite (2026-10-21). Song (empirical) finds that one same-model plus one cross-model review covers 56.7% of planted errors against 42.7% for two same-model reviews (p =.006). But Song's "same-model" reviewer is the author's model. Against two reviews by the strong cross-model reviewer, which is the arm comparable to this paper's homogeneous gpt-5.6-sol panel because neither reviewer wrote the code, the mixed pair gains only +4.1pp, CI [−2.1, 10.3], not significant. The two studies agree: mixing in a different model has not been shown to beat doubling an equally strong reviewer that is not the author. Song's significant gain comes from replacing a second review by the author's own model.
• Harness. copilot-cli beats Claude Code's /review for both Opus models tested. Opus 5 under Claude Code spawned 9.0 sub-agents per run, against 0 under copilot-cli and 0.1 for Opus 4.8 under Claude Code. Fig. 2b puts that configuration at roughly $20 per task for slightly lower recall than $2.97 under copilot-cli. This is third-party confirmation of Anthropic's own note that Opus 5 "delegates to sub-agents more readily". Discount the harness verdict for the Microsoft COI; the delegation count stands regardless of who measured it.

Reconciling the review cluster's recall numbers#

Four instruments now report a "how much does an AI reviewer catch" figure, and they range across a factor of four:

Treat these as construct differences, not contradictions. Recall falls as the ground truth widens from "the one bug that mattered" to "everything a reviewer said", and it falls on bugs that already beat a human. The safe sentence is narrow. On a single, human-confirmed, functional bug in a mid-sized open-source PR, a frontier reviewer surfaces it in its top five roughly three times in four. No figure here is a deployment escape rate, and none measures what the extra findings cost the human who reads them.

What it means for practice#

The authors' recommendations follow from the regressions:

1. Scale review effort with PR complexity. Use a cheap configuration for small, homogeneous diffs, and a strong model, high effort, the comprehensive prompt and a careful human for large or mixed ones. Splitting PRs helps both kinds of reviewer.
2. Steer the agent toward what matters to the project. It defaults to severe functional bugs, so name usability, portability, compatibility or maintainability explicitly when they matter.
3. Supply repository knowledge. This matters most for proprietary monorepos, where the bug is rarely in the diff.
4. Re-benchmark at every model change. Harness tricks expire (the comprehensive prompt's gain vanished on Opus 5), and costs can jump without warning (Claude Code plus Opus 5). The authors suggest running their mining recipe on your own repository history.

The human role they leave is the one Review as the Control Point predicts, and the one where humans measurably outperform agents: maintainability, project values, and the final accountable decision. Verification as the New Bottleneck gets its first benchmark-grade miss rate. Even the best agent at a generous k misses about one in five bugs that humans caught in review, and about a third of the bugs that shipped.

Connections#

• Same-Model Review Blindness — an independently constructed label set, but not the experiment that page needs. CRBugBench is what Greptile's case-study lacks: human-confirmed bugs, a validated cross-family judge, published κ. But every author here is human, so there is no same-model arm. Its one lineage-adjacent result, a mixed-family ensemble no better than a same-model one, concerns the reviewer's composition, not the author's lineage.
• Deterministic Engineering for Agent Code Review — the opposite construct on the same task. AACR-Bench penalises every unmatched comment, while this benchmark penalises nothing but a missed bug. That explains much of the 20% against 72% gap. Claude Code's /review comes last again, now against copilot-cli instead of a custom pipeline.
• Agent Review Comment Resolution — adoption against detection. That page measures whether humans act on agent comments; this measures whether the comment that mattered was ever written. The post-merge track is the nearest thing in the corpus to "would the agent have caught a defect that shipped".
• Closed-Loop AI Review — the population census this benchmark cannot see. All of CRBugBench's authors are human, while the census counts AI reviewing AI-authored PRs at scale. The question "how good is AI review of AI code" therefore still has no bug-grounded answer.
• Risk-Tiered Auto-Approval — the paper's conclusion is that page's gate. Lights-off only for narrowly scoped, low-consequence PRs. Detection odds fall 41% per standard deviation of log diff size, which is benchmark support for a size ceiling.
• Review as the Control Point — automated-reviewer capability, the theory's second moderator, gets a bug-grounded recall figure, a list of conditions that move it, and a measured split of strengths (humans on maintainability, agents on severity and reliability).
• Harness Shrinkage as Models Improve — the comprehensive prompt helps Opus 4.8, gpt-5.5 and gpt-5.6-sol by 4.7–5.8 points and gives Opus 5 nothing.
• Verification as the New Bottleneck — a benchmark miss rate for the automated half of verification, and the authors' own reason the human half cannot be removed yet.
• Cross-Model Review Complementarity — the controlled same/cross review experiment, reconciled with this page's mixed-family null. Song's significant mixing gain is measured against the author's own model reviewing twice. Against a strong non-author reviewer doubled, which is the comparison this benchmark ran, the gain is +4.1pp and not significant
• Weak-Verifier Ensembling — a five-way same-model ensemble buys +4.65 recall for 5.5× cost, and a cross-family ensemble does no better than a same-family one.
• Multi-Agent Collective Intelligence — sub-agent orchestration hurts on one track and helps on the other, and every multi-agent configuration is Pareto-dominated by a prompt change.
• LLM-Judge Validation — a review benchmark that reports its judge validation: three families, majority vote, κ = 0.89 against humans on 40 samples, Fleiss κ = 0.95. That is a small validation sample, and Fleiss κ across judges is reliability, not validity.
• Failures That Look Like Success — the Selenium and Elasticsearch cases, where a fluent review endorses the defect because the PR description says it is correct.
• Claude Opus 5 — best reviewer in the study and least helped by prompting. It is also the model whose delegation habit multiplied cost under Claude Code.
• Claude Code — the alternative harness, beaten by copilot-cli in a comparison run by Microsoft authors.
• Writer/Reviewer vs Agent-to-Agent Review — its Axis 2 (who the reviewer is) gets a reviewer-diversity datum, and Axis 4 gets a third recall construct.

Open Questions#

• Is the 82.1% human-plus-agent union reachable in deployment? It is computed by overlaying two independent detections. The study it cites (Tufano et al., ICSE 2025) found LLM reviews redirect human attention toward low-severity issues without raising high-severity detection. A study of human reviewers working with the agent's review against working without it, scored on CRBugBench-style ground truth, would show whether collaboration recovers the union or a lower figure.
• How much of the open-source vs proprietary gap is familiarity rather than difficulty? Diff visibility explains 20%. The open-source repositories were created before 2025 and the tasks run 2015–2026, so models may have seen the fix commits. Splitting the open-source tasks by date relative to each model's training cutoff, using the calendar-time covariate the regression already has, would separate memorisation from the knowledge-gap explanation the authors favour.
• Does supplying repository knowledge close the proprietary gap, and at what precision cost? The paper recommends knowledge bases and memory but tests neither, and its metric cannot see the extra false positives that more context and longer reviews might produce. The test is an ablation on the 800 proprietary tasks with and without retrieval over design docs and past reviews, scored on recall and a precision measure.

Sources#

• (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection — Chenyang Yang (CMU; work done at Microsoft Research), Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill (Microsoft), Carmen Badea and Christian Bird (Microsoft Research), (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection, arXiv 2610.13668, 2026-10-09, 22 pp. empirical, tier kept. Sections used: §1, §3.1–3.5 (pipeline, counts, validation), §4.1–4.2 (design, threats), §4.3–4.6 with Figures 2, 5, 6, 7, 8 and Tables 1–3, and §5.1–5.3. Parse warning, in the measuring-beyond-accuracy-saturation convention: PDF-derived (docling 2.126.0, docling_mlx 0.1.1). Ingest verify warned table-collapse and table-weld on Tables 1 and 2. Both were badly scrambled in the raw markdown: multi-value cells, the claude-opus-4.8 and 4.6 rows welded together, the gpt-5.4, claude-sonnet-5, gpt-5.6-luna and claude-haiku-4.5 rows shifted by a cell or more, and Table 2's Fix/Cmt labels misaligned. Every Table 1 and Table 2 figure on this page was re-read from pdftotext -layout on the local PDF, not from the raw, and the headline arithmetic was checked against the prose (71.9 = mean of 79.6 / 64.2, 47.7 from Opus 4.6, $2.97, 78% at 21% cost, +5.8 at 1.10× = Opus 4.8, +5.3 = gpt-5.6-sol, +2.8 at 3.68× = mean over Table 3's five configurations, +4.7 at 5.55× = Ensemble@5). All of it reproduces. Table 3 parsed cleanly. Figures 2 and 5 were viewed. The proprietary-portion values (~53 for Opus 5, ~37 for gpt-5.6-sol) are read off Fig. 2a by eye, and the text's "up to 25.5%" could not be tied to a specific value. The PDF's "Received 20 February 2007" footer is an ACM template artifact.