Cross-Model Review Complementarity

Sources#

• When Does a Second Model Help? Cross-Model Review in LLM Verification

Summary#

The corpus's lineage evidence for review so far is one vendor's crossover (Same-Model Review Blindness, Greptile, case-study), a mixed-family review ensemble that did no better than a single-model one (Code Review Agents on Human-Verified Bugs (CRBugBench), CRBugBench), and a stack of judge-side correlation audits (Cross-Model Error Entanglement). Tae-Eun Song (Daejeon Jungang Cheonggua Co., arXiv 2610.01471, v1 2026-10-01, v2 2026-10-06) adds the first controlled experiment that holds the author fixed and varies the reviewer across same-model and cross-model conditions, with a planted-error ground truth and a pre-analysis audit of its own records.

Ten conditions × 90 sessions = 900 review sessions. One generator, Claude Opus 4.6 (per the run logs), wrote 30 artifacts in Korean: ten Python modules, ten technical tutorials, ten presentation scripts. The same model then injected five errors into each, one per type (FACT, CONS, CTXT, RCVR, MISS), giving 150 planted errors. Reviewers were the generator itself in four configurations, plus three external models, each with and without the original request:

The primary test (Wilcoxon on 30 artifact-level means against CCR, Holm over nine comparisons) finds only SR2 different (p =.011). Everything else, including the top cross-model reviewer, is p = 1.00 after correction. The author says the right thing about that: no detected difference is not evidence of equivalence.

Evidence note. empirical, tier kept. It is a controlled design with paired tests, multiple-comparison correction, a bootstrap CI on the headline union, and an unusually candid audit. Before analysis the author excluded one baseline run (SR run 3) because 92% of its finding text appears verbatim in the script that wrote the files, and 14 failed cross-model calls that had been scored as empty reviews. Both versions of every headline are reported (Appendix F). The weaknesses are stated by the paper and are serious. It is a single-author, non-peer-reviewed preprint built on the author's own three earlier preprints, and those supply all four same-model baselines, which are unverified: they were transcribed by the host session and carry no raw model output. The ground truth was planted by the generator and never independently annotated. The keyword-and-line matcher was never hand-audited, and a location match alone clears its threshold. All artifacts are Korean, and 20 of 30 run-1 CCR reviews came back in English, which the Korean keyword matcher may under-credit. GPT-5.4 ran in Codex CLI with file access to an unrecorded working directory. Each capability tier is one model. Data is "available on request". No vendor COI; Claude Code wrote the analysis scripts and drafted much of the text, as disclosed.

What it finds#

1. A single cross-model review is not significantly better than a fresh same-model one. XMR-GPT 32.3% against CCR 28.6% (adjusted p = 1.00; run-1 t-test p =.478). The prompt-matched comparison, XMR-GPT-R at 32.1%, is the same. On recall the top-tier reviewer is well ahead, 38.4–39.8% against 27.1%, and it pays back most of that in precision. The paper tests F1 only, so the recall gap is untested.

2. The two find partly different errors. On run 1, CCR matched 64 planted errors and XMR-GPT-R 56, with 35 in common: Jaccard 41.2%, so about 59% of the union comes from only one reviewer. Finding-level overlap across all run pairs is 52–62% within a model (CCR 52.2%, SR 61.8%, XMR-GPT 59.3%, XMR-Ge 55.1%) and 37–46% across models (CCR vs XMR-GPT 46.1%, vs XMR-Ge 36.5%). The author treats this as descriptive, because lower overlap also follows mechanically from different detection rates and finding counts.

3. At two review calls, mixing beats doubling the author's own model. One CCR plus one XMR-GPT-R review matches 85/150 (56.7%) of planted errors. Two CCR reviews match 64/150 (42.7%). That is +14.0pp, with a bootstrap 95% CI of [6.7, 22.0] and Holm-adjusted p =.006. Self-review as the same-model partner gives the same sign (SR + XMR-GPT-R 77/150 against SR + SR2 53/150, +16.0pp, p =.008). These are set-level recalls over run 1, and run 1 is CCR's best run (56, 37 and 29 matched errors over three runs). Across runs, mixed unions cover 44.7–56.7% and CCR-only unions 30.0–42.7%.

4. It does not beat doubling the cross-model reviewer. Against two XMR-GPT-R reviews (29 artifacts without a failed session, 76/145 against 82/145), the mixed pair gains 4.1pp, CI [−2.1, 10.3], adjusted p =.184. So the experiment cannot say whether the union gain came from model difference or from adding a better reviewer. The author says so in the abstract.

5. Reviewer capability moves F1 more than any independence dimension. XMR-GPT exceeds XMR-Ge by 8.3pp (unadjusted p =.020, outside the test family). That is larger than the whole same-model spread (6.9pp) and far larger than the CCR–SR gap (1.5pp). The lightweight cross-model reviewer, at 24.0–27.0%, is at or below same-session self-review. Model independence with a weak reviewer bought nothing.

6. Errors stay correlated across models. The observed unions fall below the independence reference r_A + r_B − r_A·r_B, both pooled and within artifact. So the non-overlap in (2) is real but smaller than independent detectors would give. A third model adds little: CCR ∪ XMR-GPT-R ∪ XMR-GePro-R reaches 58.0%, only +1.3pp over the two-model union.

Where the sign reverses: code against documents#

On documents, all six cross-model conditions (30.1–42.1%) score above all four same-model ones (17.4–24.5%), the cleanest descriptive pattern in the paper. On code the same-model fresh-session reviewer leads. But the lead is precision, not recall: on code, CCR's precision is 46.9% against 35.6% for XMR-GPT, while its recall is lower, 36.7% against 40.0%. So on recall the code result has the same sign as Greptile's (the cross-family reviewer catches more), and it is a recall-only instrument like Greptile's that would show it. Category estimates carry no intervals and were not tested.

By error type (per-session recall), the top-tier cross-model reviewer leads CCR on FACT (58 vs 48), CONS (53 vs 37), CTXT (26 vs 13) and RCVR (34 vs 19). RCVR varies most across the six cross-model conditions (7–38%), which fits it being capability-sensitive.

A confound the paper does not name (wiki reading). The planted errors were injected by the generator, in its own sessions. A same-model reviewer is therefore looking for errors its own model chose and wrote. That could help it (it recognises its own edits) or hurt it (blind to what it found plausible enough to plant). Either way, the "same-model" arm here is not reviewing naturally occurring authoring errors, which is what Same-Model Review Blindness measures.

The other independence dimensions#

The paper frames reviewer independence as four binary dimensions: context (C), temporal (T), structural (S) and model (M). Information restriction (R), withholding the original request, cuts across them. It is the vocabulary Optimizer–Evaluator Decoupling has been assembling one axis at a time, and here the axes come with numbers on one task:

• Context reset alone is worth almost nothing measurable. CCR against SR is 28.6% against 27.1%, a 1.5pp gap with p =.26. That comparison also changes whether the request is visible. Overlap between CCR and SR findings is 59.9%, as high as within-condition overlap: "we see no clear sign that changing the context … changes which errors the model finds." The author's own earlier CCR preprint had claimed a significant gain. That claim rested on run 1 alone, and with the provenance-uncertain run excluded and all runs used, it does not hold. The paper says so itself.
• A second pass in the same session is harmful. SR2 is the only condition significantly below CCR (21.7%, p =.011). This is consistent with the earlier D-CCR report that repeated rounds raise false positives by 62% without recall gains.
• A sub-agent with the request does no better (SA 23.8%, not significant).
• Withholding the request helps weaker cross-model reviewers, as untested point estimates. Lightweight +3.0, capable +2.2, top-tier −0.2. With failed sessions scored as zero these become +0.9, +2.2 and +0.9, so the capability gradient is not robust. Restriction also lowers "model bias" false positives slightly (Flash 13% → 10%, Pro 10% → 8%).

A false-positive class cross-model review adds#

Of 1,995 cross-model false positives, 187 (9.4%, keyword-classified, unvalidated) flag things from the reviewer's own training rather than real defects. Most of these (136/187) call generator-specific features or tools nonexistent. Severity labels are a weak triage key. Cross-model reviewers marked 70.0% of true positives and 54.7% of false positives Major or Critical; same-model reviewers marked 54.5% and 28.2%. A cross-model reviewer is louder about its mistakes. Since a finding of a real pre-existing defect also counts as a false positive, some of these may be real.

Outside software, and the external check#

Of five case studies, only B3 (12 PolitiFact articles, 43 errors, 108 reviews) discriminates. XMR-GPT again has the highest F1 (24.4% against CCR 20.7%, untested), and the CCR ∪ XMR-GPT union reaches 86.0% recall. That union pools six reviews against three and is not call-matched. B1 and B2 hit 100% under every condition. B4 (reviewing the paper's own plotting code) and B5 (a security audit of two C libraries across three models × three perspectives) have no ground truth. In B5, mean inter-model Jaccard is 0.152 against 0.228 inter-perspective, so changing the model diversifies findings more than changing the prompt. Nobody checked whether those findings were correct.

On ReaLMistake's public detector outputs (Appendix D), the mixed pair gains +0.5pp, CI [−1.2, 2.2]. The analysis plan was committed before computing. In every responder cell the false-positive rate moved with recall, which looks like a shift in operating point rather than added coverage. The unit, domain and 2024-era detectors all differ, so the paper reads this as neither replication nor contradiction. It is the one test outside the author's own data, and it found nothing.

Reconciling with the rest of the review cluster#

• CRBugBench's mixed-family ensemble did no better than a same-model one. Song's mixed pair beat a same-model pair by 14pp. These agree once you see which arm is the baseline. Song's significant gain is against two reviews by the author's own model. Against two reviews by a strong non-author model, the gain shrinks to +4.1pp and is not significant. CRBugBench's homogeneous arm is three gpt-5.6-sol candidates reviewing human-authored PRs, so no reviewer is the author, and its comparison is the analogue of Song's XMR-GPT-R × 2 arm. Both results then say the same thing: on these instruments, mixing a different model into a review pair has not been shown to beat doubling an equally strong reviewer that is not the author. Two differences in method remain. CRBugBench consolidates candidates through an orchestrator into a ranked top-5 rather than taking a raw union, and its effect is within a point and untested. Song's +4.1 is in the mixing direction with a CI that includes zero.
• Greptile: different design, and the stronger one on the question both ask. Greptile's crossover fixes reviewer capability by construction: both reviewers average about 56%, so the 8.9pp is what survives reviewer and dataset main effects. Song cannot do that. With one author and one model per tier, every cross-model contrast changes capability too. So on whether lineage itself matters, the lower-tier source has the better design, and this source corroborates its sign only on recall and only with the top-tier reviewer (38.4–39.8% against 27.1%). On F1 and with weaker reviewers it does not. Neither supersedes the other. Greptile measures natural authoring bugs with a vendor-built label set; Song measures self-planted errors with a golden-by-construction label set.
• Cross-Model Error Entanglement predicts the sub-independence unions. Kohli and Kuai find cross-family error correlation only modestly below same-family. Song's cross-model overlap is lower than within-model overlap by about 11pp, and the unions still fall below the independence reference. This is a third instrument, on detection rather than judging, finding "different, but not independent". The paper's own citation is Kim et al. (ICML 2025): on HELM, two models that both err pick the same wrong answer ~60% of the time, and more accurate models are more correlated.
• Weak-Verifier Ensembling: a third reviewer adds 1.3pp. That fits the pooled-dependence ceiling that page records. Diversity buys the first increment and then flattens.
• Writer/Reviewer's "fresh context, no bias toward own code." CCR is that pattern, and it is measured here: a fresh session with the artifact only is 1.5pp (ns) above same-session self-review and finds largely the same errors (59.9% overlap). This supports that page's argument that a /clear does not buy reviewer independence. It now rests on a measurement, not only on Greptile's lineage result.

What it means for practice#

The author's guidance, which he labels tentative and limited to this sample:

1. For coverage, pair a same-model review with a cross-model one rather than running the same model twice. This is the result with a significant test behind it. It is also the shape of OpenAI's codex-plugin-cc (Codex reviewing from inside Claude Code), which the paper cites as industrial cross-model review tooling, and of DHH's Opus-authors, Codex-reviews routine.
2. For a single review, use the strongest reviewer available. Its family matters less: top-tier cross-model scored highest, though not significantly above CCR.
3. Never use a lightweight cross-model reviewer as the independent check. It scored at or below same-session self-review. "Route to the other vendor" is only as good as the other vendor's model at the tier you route to.
4. Do not re-review in the same session. It was the only significantly worse condition.

What it means for Verification as the New Bottleneck: even the best two-call union matched 56.7% of planted errors under an automated matcher, so the paper's own closing line is that the framework augments human review rather than replacing it.

Connections#

• Same-Model Review Blindness — the vendor crossover this experiment can corroborate only in sign and only on recall. Its design fixes reviewer capability, which this one cannot, so on lineage-versus-capability the case-study source has the stronger design
• Code Review Agents on Human-Verified Bugs (CRBugBench) — the mixed-family ensemble null, reconciled above. Both instruments say mixing a non-author model into a pair has not been shown to beat doubling an equally strong non-author reviewer
• Cross-Model Error Entanglement — sub-independence unions and 37–46% cross-model overlap, a detection-task measurement of "different but still correlated"
• Weak-Verifier Ensembling — a third reviewer model adds 1.3pp, the diminishing-returns shape that page's dependence ceiling predicts
• Optimizer–Evaluator Decoupling — the four-dimension independence framework (context, temporal, structural, model, plus information restriction), with context reset measured at ~0 and model independence confounded with capability
• Closed-Loop AI Review — the controlled same/cross experiment that census's authors call for, in miniature: one known generator model, same-model and cross-model reviewers, planted ground truth, scored on recall and false positives. It lacks a related-model arm on the author's side and uses no PRs
• Writer/Reviewer vs Agent-to-Agent Review — Axis 1 gets its first ablation on review (context reset, request withholding), and Axis 2 gets an empirical number with a capability confound
• Reference-Free Judge Over-Crediting — the ceiling argument: reviewers from different families still share a plausibility basin, and here the cross-model unions fall short of independence
• LLM-as-a-Judge — the 9.4% "model bias" false positives are the judge-side failure where a reviewer grades against its own training rather than the artifact
• Verification as the New Bottleneck — 56.7% union coverage of planted errors as an automated-review ceiling on this task
• Stopping Under a Noisy Verifier — SR2: a second same-session pass is significantly worse, so a verify-repair loop that re-asks the same session buys false positives, not recall
• LLM-Judge Self-Preference — the scoring-side counterpart of this page's "model bias" false positives. Arena's OpenAI judges hold a taste that disagrees with both people and other labs' judges (they agree with the human voter less than half the time and least with the other AIs), the preference-task version of a reviewer grading against its own training
• Claude Code Best Practices — the Writer/Reviewer rationale ("fresh context avoids bias toward own code") this experiment tests: CCR vs SR differs by 1.5pp F1, not significant, which is part of why that page's reviewer-lineage question was reopened as a partial on 2026-10-21

Open Questions#

• Does a capability-matched cross-family reviewer still beat two same-model reviews? Song's union gain is significant against doubling the author's model and not against doubling the top-tier reviewer, and each tier is one model. The discriminating arm is a cross-family reviewer chosen to match the generator's single-review F1 (or a same-family reviewer of matched strength from another lineage), run against CCR × 2 with the same matcher. A gain that survives capability matching is lineage. One that vanishes was capability all along.
• Is the same-model edge on code an artifact of self-planted errors? The generator injected every error, and on code the same-model reviewer's advantage is entirely precision (46.9% vs 35.6%) with lower recall. Rerunning the code arm with errors planted by a third model or a human, or on naturally occurring bugs, would show whether the reversal survives.
• Does withholding the request help cross-model reviewers once failed sessions are handled consistently and the effect is tested? The +2.2/+3.0 gains shrink to +0.9 for two tiers under zero-scoring, and CRBugBench separately finds reviewers trusting the PR description over the code. A paired test of description-visible against description-withheld review on one ground truth would settle whether the request is a help or a hijack surface.

Sources#

• When Does a Second Model Help? Cross-Model Review in LLM Verification — Tae-Eun Song (Daejeon Jungang Cheonggua Co., Ltd.), When Does a Second Model Help? Cross-Model Review in LLM Verification, arXiv 2610.01471, v1 2026-10-01, v2 2026-10-06 (v2 adds the provenance audit, exclusions, Appendix D and corrected Table 1 labels after a PREreview of v1), 16 pp. empirical, tier kept. Sections used: abstract, §1–3 (framework, conditions, hypotheses), §4.1 (design, provenance audit and exclusions), §5.1–5.9, §6.2, Limitations, Appendices A (matcher), D (ReaLMistake check), F (zero-scored values), G.1, H (case studies). Parse note: PDF-derived (docling 2.126.0 / docling-mlx 0.1.1, confidence grade excellent). Tables 2, 3 and 4 have one value per cell with row counts matching the conditions the prose describes. Every Table 2 value cited is restated in the §5 prose, and Tables 3 and 4 were checked cell-for-cell against pdftotext -layout on the local PDF, with no discrepancy. Figure 1 is a Venn diagram whose caption gives all its numbers (85, 35, 65), and Figure 2 is a bar chart of Table 2, so neither was opened. The recall and precision columns cited from Table 2 are per-session means, so F1 is not their harmonic mean (Appendix A says so).