Deterministic Engineering for Agent Code Review

Sources#

• Claude Code Changelog
• How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes
• OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review

Summary#

Li, Zhang, Wu et al. (Alibaba Group / Nanjing University / Peking University, arXiv 2608.09290, v1 2026-08-10, empirical) build OpenCodeReview on a stated design philosophy — deterministic engineering for uncertain agents — and measure it against two shipped products on a repository-level review benchmark. The philosophy is one sentence: rather than granting the agent maximal freedom and hoping it converges, inject determinism at chosen points in the pipeline. Three points, each aimed at a named source of variance:

1. Rule-Guided Dispatch — which files get reviewed, and against what criteria, is decided by a four-tier glob-matched rule chain, not by the agent. "The same PR always yields the same file and criterion assignment."
2. Grounded File Review — no general shell. Six review-specific tools, each with a hard output cap, exposed through a ReAct loop, one parallel SubAgent per file.
3. Independent Reflection — a falsification-first filter that can delete comments and never write them, run by the same LLM under a deliberately narrower information boundary than the agent it audits.

The headline is a 2.17× SEM-F1 improvement at 5–15× fewer tokens. The direction underneath it is the part to carry, and it is not what the abstract advertises: this is a precision-for-recall trade, bought at a large discount. Precision rises 4.7× and recall falls, so the winning system surfaces fewer genuine issues in absolute terms than the unconstrained baseline it beats.

This is the corpus's second empirical source arguing determinism-beats-autonomy for agent design, after Deterministic Pre-Execution Gates. Neither runs the arm that would isolate the claim.

The result, and the direction it runs#

AACR-Bench (Zhang et al., arXiv 2601.19494): 200 real pull requests from 50 repositories across 10 languages, with 1,505 ground-truth review comments validated through three rounds of cross-validation by over 80 senior engineers. Comments are scored by semantic matching — an LLM judge decides whether a generated comment expresses the same concern as a ground-truth comment at the same location — yielding Precision, Recall, and their harmonic mean SEM-F1.

The single comparison the paper's headline rests on, Table 3's best row against the same model under the strongest baseline:

	SEM-F1	Precision	Match/Gen	Recall	Match/GT	Avg. tokens	Avg. time
OpenCodeReview (Claude-4.6-Opus)	25.10%	33.90%	301/889	20.00%	301/1505	385K	1m23s
Claude Code /code-review (Claude-4.6-Opus)	11.57%	7.23%	435/5980	28.90%	435/1505	5,664K	13m06s

Read the middle four columns before the outer ones. The constrained system:

• wins SEM-F1 2.17× and precision 4.69×;
• loses recall, 20.00% against 28.90% — the baseline finds 435 of the 1,505 expert-verified issues and OpenCodeReview finds 301, so 134 real issues are surfaced by the system that scores half the F1 and by nothing else;
• spends 14.7× fewer tokens (385K vs 5,664K per sample) and 9.5× less wall clock (1m23s vs 13m06s).

The paper states the trade plainly in §4.2 — "Claude Code achieves its highest recall (28.90%)… but at the cost of severe precision degradation (7.23%)" — and then does not carry it into the abstract, the introduction, or the conclusion, all three of which lead with the 2.17× and the token saving. A reader who takes only SEM-F1 concludes the constrained system is twice as good at review; the counts say it is twice as good at not wasting your attention, and worse at finding things. Which of those you want is a deployment decision the metric does not make for you.

The efficiency figures generalize across the panel where the quality figures are more modest. Token ratios reconstructed across the five paired Claude Code rows run 5.4× (GLM-5.1) to 14.7× (Claude-4.6-Opus), which is what licenses the abstract's "5–15×"; SEM-F1 ratios over the same pairs run 1.27× to 2.17×, matching §4.3's "1.3–2.2×". Against Codex the story is different in kind: 21.00% vs 8.36% SEM-F1 at comparable cost (422K vs 525K tokens, 2m51s vs 2m58s), because Codex's single-agent /review loop terminates early — 266 comments generated across 200 PRs, 74 matches, 4.92% recall, the lowest of any configuration by a factor of two.

The frontier is a frontier, and the caption oversells it#

Figure 3 plots all 12 configurations in precision–recall space (axes: Precision 0–40%, Recall 0–30%, both linear, no multiplier; viewed per the image two-pass rule). What it shows, and the prose does not:

• The upper-right quadrant is empty. Best precision anywhere is 37.80% at 11.70% recall; best recall anywhere is 28.90% at 7.23% precision. Nothing in the study achieves both above 25%. On this benchmark the two systems are not ranked, they are two operating points on a frontier no configuration escapes.
• Model choice moves you along the frontier, not off it. Within OpenCodeReview's own six rows, the best-precision configuration (Claude-4.8-Opus, 37.80%) is tied for the worst recall (11.70%), while the best-recall configuration (Claude-4.6-Opus, 20.00%) sits 3.9pp lower on precision. The system's design fixes a region; the backend picks a point inside it.
• Figure 3's own caption overstates the result. It says Claude Code and Codex "occupy distinct regimes with lower precision at comparable or lower recall." Two Claude Code rows have recall (28.90%, 23.37%) above every one of OpenCodeReview's six. §4.4's prose is careful about this and the caption is not; quote the prose.

What the comparison cannot isolate#

There is no ablation in this paper. No arm removes reflection, no arm replaces rule-guided dispatch with agent triage, no arm swaps the bounded tool set for a shell, no appendix, three tables total. The only comparison of any kind is the full pipeline against two entirely different products.

That matters more than a missing-ablations note usually does, because the paper's central claim is causal and specific: §4.2 attributes the 2.17× "solely to system design," and §5 concludes that "system design contributes more to review quality than model choice." What the design actually rules out is model advantage — all systems share the backend, which is real and well done. What it cannot separate is the three deterministic mechanisms from each other, or any of them from every other difference between three independently-built codebases: prompt text, dispatch logic, comment-line anchoring, output formatting, termination criteria, and each vendor's post-training.

Two things sharpen this into something more than a generic caveat.

The baseline is a pinned older product, and this vault can date how old. The paper benchmarks Claude Code v2.1.169, characterizing it in §4.1.2 as "a general-purpose agent loop without review-specific tools or file-level parallelism," and defends its two-baseline design in §6 on the grounds that both "represent the current state-of-the-art in industry-shipped coding agents." Against the vault's own snapshot of the Claude Code changelog (Claude Code Changelog, vendor-claim, rolling document snapshotted 2026-08-03, scoped v2.1.200–2.1.220), that is not what shipped. By v2.1.202 the release notes read: "Changed /review <pr> back to a fast single-pass review; use /code-review <level> <pr#> for the multi-agent review at a chosen effort level" — an explicitly multi-agent, effort-tiered /code-review, at least 33 releases after the version under test and before this paper was submitted. v2.1.206 then records "Improved /code-review findings quality on claude-opus-4-8 across all effort levels" — a vendor change aimed at exactly the backend of the paper's second Claude Code row. Later still, v2.1.215 stopped Claude self-invoking the skill and v2.1.218 moved /code-review into a background subagent. Nothing here says the paper's numbers are wrong; it says the characterization is stale — "no file-level parallelism" is a property of v2.1.169, not of the product class — and the fair reading of "state-of-the-art industry baseline" is materially weaker than §6 claims. This is Same-Model Review Blindness's warning arriving from the other side: a number measured through a vendor's /review command is a measurement of one pinned build of a product, not of a model or of an architecture.

The comparison is confounded on exactly the axis the paper wants to isolate. Both baselines route through a stock vendor review prompt, and Same-Model Review Blindness measured what that prompt does to output volume: Codex reviews land at 1–2 comments and Opus at 7–8 through their own /review features, with a single instruction to target 7–10 comments enough to change which findings get posted. Comment volume is the term that sets precision here (5,980 generated vs 889), and it is a knob in the baselines' prompts that this study never touches. So the strongest single number in the paper — 4.7× precision — is confounded with a vendor prompt-tuning decision about noise tolerance.

(§4.2 also contains an arithmetic slip: it puts Claude Code's comment volume at "4,580 across 200 PRs" where Table 3's Match/Gen for the same configuration is 435/5980. The table is the consistent one — 435/5980 = 7.27% reproduces the printed 7.23% precision, 435/4580 = 9.50% does not. Cite 5,980. Similarly §4.1.2's "Claude Code is evaluated with six models, excluding GPT-5.5" describes five rows, which is what Table 3 and Figure 3's twelve points contain.)

The reflector sees less, on purpose#

The most transferable mechanism in the paper is also its least-evidenced. The reflection module runs after a SubAgent finishes and receives only the file's diff and the SubAgent's comments — not the tool-augmented exploration the SubAgent used to form them. The reflector is the same LLM; only the information boundary differs, and it is asymmetric in the direction nobody usually builds: the auditor sees strictly less than the author.

The design follows from that asymmetry rather than fighting it:

• Falsification, not verification. The reflector's only question is whether the diff contains direct counter-evidence to a comment's key claim. A comment resting on context outside the diff is explicitly not flagged, because the SubAgent may hold evidence the reflector cannot see. Under-informed auditors normally over-reject; scoping the veto to what the auditor can decide is the fix.
• Filter-only. The module deletes and never generates, so "reflecting on a set of comments can only reduce false positives, never introduce new false ones." Issue breadth is set by the SubAgent alone; per-comment reliability by the reflector alone.
• Fail-open on a parse error. Unparseable reflector output retains every comment, "prioritizing recall over precision in the failure case."

That last property is the same discipline Deterministic Pre-Execution Gates adopts for its predicates — a gate that raises an exception is logged and the call proceeds — and for the same structural reason: a broken filter must degrade to no filter, never to a new failure mode. The directions differ (a failed gate lets a risky write through; a failed reflector lets a hallucinated comment through) and the invariant is identical.

Where the evidence stops. §5 concludes that "for reflection, the information boundary may matter more than the model identity," and the entire support offered is the system's precision range (25–38%) across six backends. That is not a test of the claim. Testing it needs at minimum a no-reflector arm, a Reflexion-style same-model/same-context arm, and a different-model arm; none was run, and the module's contribution to the 2.17× is unknown. The vault holds a measurement pointing the other way on this very task: Same-Model Review Blindness varies only model lineage on code review, holding harness, diff and ground truth fixed, and finds 6–12 points of high-severity recall riding on it. Weighted honestly — that source is case-study with vendor-built ground truth, this one is empirical — the resolution is not that either wins but that the paper's tier does not extend to this particular sentence: model identity is the axis with a number attached, and information boundary is the axis with a design argument. Both remain live.

Bounded at the tool, not compressed after#

The action-space constraint is a concrete artifact worth copying: six tools, each with a published ceiling.

Tool	Purpose	Bound
file_read	read a file by path and optional line range	max 500 lines/call
file_find	locate files by name keyword	max 100 results
code_search	search text patterns across the repository	max 100 matches, 10s timeout
file_read_diff	view another changed file's diff	pre-computed diff
code_comment	submit a review comment	parsed asynchronously
task_done	signal completion	terminates the loop

The stated purpose is to prevent any single call from saturating the window — the token snowball the paper cites SWE-Effi and Efficient Agents for. This is the prevention-side answer to the problem Tool-Output Pruning attacks on the compression side, and the comparison is the useful part. A learned pruner reads the observation and decides which ~30% of lines to keep; a hard cap truncates at line 500 and is entirely content-blind, so it can discard the one function body the agent needed while a probe-driven head would have kept it. What the cap buys instead is that it costs no model call, no extra prefill, and no prefix-cache break — the three costs that made most pruners in that page's seven-method comparison lose tokens overall, and the cache term that is still unpriced there. Neither source runs the other's arm, so "cap it or compress it" has no measurement anywhere in the corpus.

Two further bounds run alongside, both keyed to context-window fraction rather than absolute tokens (Context Window Smart Zone): files whose diff exceeds 80% of the model's context window are filtered out before dispatch, and in-loop compression triggers asynchronously at 60% utilization and synchronously at 80%, summarizing the middle of the history while freezing the system prompt and the most recent turns. The ReAct loop is capped at 30 iterations, with an empty-round detector that prompts the agent to act or terminate after three consecutive tool-free rounds.

The comment-anchoring stage is the pipeline's other quietly deterministic move: rather than trusting the agent's line numbers, a three-stage fallback matches the agent's existing_code snippet against the diff's new-side hunks, then the full file, and only then falls back to an LLM relocation step. Two deterministic attempts before a model call — the same ordering Risk-Tiered Auto-Approval uses at the merge gate.

Every metric here is one unvalidated judge#

Every number on this page is a Qwen3-235B-A22B-Instruct semantic-match verdict. §6's mitigation is to run the judge five times per configuration and report the mean, and its defense is that "since all systems are evaluated by the same matcher under the same protocol, relative comparisons remain valid."

That defense is precisely the move LLM-Judge Validation names: same-matcher consistency is not matcher validity. No agreement statistic, no Cohen's κ against a human panel on the matching decision, no position or order controls on the pairwise match, no per-run variance from the five samples. And five samples of one judge is a homogeneous jury, which that page prices: measured intra-class error correlation of ρ ≈ 0.66–0.97 means repeated sampling of a single judge buys far less than independence would predict, so the five-run mean is a much weaker variance control than it looks.

Two mitigations are real and should be credited. AACR-Bench's ground truth is expert-verified — three rounds of cross-validation by 80+ senior engineers — which is a stronger label basis than anything else in the vault's review cluster. And §6 volunteers the construct limit that matters most: "a genuinely useful comment that does not match any ground-truth item would be counted as a false positive."

Take that admission seriously, because it changes what "7.23% precision" means. It is a match rate against a fixed reference set, not a correctness rate — it does not say 93% of Claude Code's comments were wrong. Agent Review Comment Resolution measured the human-side analogue on 54,713 real agent review comments and found the opposite shape: ~71% resolved, with only 63 of 470 argued-and-unresolved discussions factually wrong or false positives and 11 of 470 dismissed as low-value noise. Different corpora, different agents, and that sample covers only argued cases — but the two constructs cannot be pooled and the gap between them is enormous. A system optimized to raise reference-match precision is optimizing something that has never been shown to track what developers act on.

Connections#

• Agentic Code Generation as Compilation — the corpus's third determinism-beats-autonomy claim, and the third one with no ablation. Bridgewater's PAT reports ~4x this page's baseline (Claude Code) on codegen latency and near-instant re-execution after a small edit, from three simultaneous injections — a typed plan IR, DAG-scheduled mandatory validation passes, and a static-analysis caching layer — none isolated. The pattern across all three sources is now hard to miss: purpose-built pipeline beats general-purpose agent by a large multiple, credit assignment untested every time. PAT is the weakest of the three as evidence (a first-party talk, no task set, no sample size, no quality metric at all) and the most specific as design, since its determinism is a measured output property — two agents emitting identical code 95% of the time — rather than an architectural intention

• Deterministic Pre-Execution Gates — the corpus's other determinism-beats-autonomy result, and the second one to skip the same arm. Both sources constrain an agent and report a large gain (+12.4pp task success there, 2.17× SEM-F1 here) in different task domains, different years, different labs — genuine independent corroboration of the direction. And neither runs the isolating experiment: that paper never checks whether forcefully prompting its four gated rules recovers the lift, and this one has no ablation at all, so "the hard constraint is doing the work, not the instruction" remains the shared hypothesis of two papers rather than the finding of either. The mechanisms are also cousins rather than twins — that page's gate is a pure predicate over a proposed mutating call that rejects before execution; nothing here rejects anything, the determinism sits in dispatch (which files, which criteria) and in caps (how much any tool may return). The convergence worth keeping is on failure semantics: both make their deterministic layer fail open, so a bug in the constraint degrades to the unconstrained system rather than to a new failure mode

• Latent vs. Deterministic Space — Rule-Guided Dispatch is Tan's diagnostic applied to triage: which files merit attention and against what checklist is a decision the paper argues sits on the wrong side ("if left to the agent, this decision varies across runs"), and moving it into glob patterns with first-match-wins semantics is the fix. It also extends that page's third category. The rules are natural-language review checklists selected by deterministic path matching — the dispatch is deterministic and the criterion content stays latent — which is a hybrid neither Tan's seating example nor the τ²-bench gates contain: not "move the prose into code" but "let code decide which prose the model gets." The caveat is that this instance is confounded where the gate paper's was clean: there the only thing that moved was which side one rule class ran on, here three mechanisms move at once against two different products

• Optimizer–Evaluator Decoupling — a further independence axis, running the opposite direction from every other one on that page. Its existing axes all restrict what the grader may learn about the grade or reveal about itself (Bun's diff-only reviewer, SEAL's one-bit sealed audit, Cursor's decorrelated lenses, Zhou's commit-before-conditioning). This restricts what the grader may learn about the world: the reflector is the same model, on the same artifact, given strictly less evidence than the author — and the paper argues that under-information is what breaks the self-reinforcing bias, since "it sees less than the agent, not more." The design consequence is the interesting half: an under-informed auditor must be scoped to falsification only, because it cannot distinguish a wrong comment from one resting on evidence it was denied, so the veto is narrowed to claims the diff itself contradicts and the module is forbidden from generating. Weight it as a design, not a result — the module is never ablated, and §5's "information boundary may matter more than model identity" is argued from cross-product totals rather than from any arm that varies the boundary

• Same-Model Review Blindness — the two sources disagree about what an agent reviewer's recall is, by a factor of two and a half, and the disagreement is mostly construct. Greptile puts frontier reviewers at 50.5–62.0% recall on high-severity bugs; the best system here reaches 20.00% and the best baseline 28.90%. Neither is wrong: that ground truth is ~1,500 vendor-labelled P0/P1 bugs in agent-authored PRs with stylistic and documentation comments excluded, this one is 1,505 expert-verified comments of every kind across 200 human-and-agent PRs. So the number is set by what counts as a finding, and no population-level recall figure for agent review exists — the two empirical-ish measurements of it differ 2.5×. Two things do transfer. Greptile's finding that a recall number measured through /review measures a product is the independent basis for this page's staleness objection to a v2.1.169 baseline. And its 6–12 points of lineage effect is the vault's only measured counterweight to §5's claim that information boundary matters more than model identity

• Agent Review Comment Resolution — the third leg of the review layer's efficacy, and the one that deflates this paper's best number. That page and Same-Model Review Blindness already pair adoption (71.4% of comments resolved) with recall (52–62% of bugs named); this adds precision against an expert reference set (7.23% to 37.80% across twelve configurations). All three must stay unpooled, and this one carries the sharpest construct warning: §6 concedes that a genuinely useful comment matching no ground-truth item scores as a false positive, and that page's card sort found only 11 of 470 argued discussions dismissed as low-value. A 7% reference-match rate and a ~71% human-resolution rate are measuring different things, and optimizing the first has never been shown to move the second. The two also converge on where review agents actually fail: the modal genuine rejection there is project context the agent could not see (23.8%), which is exactly what this architecture spends its bounded tools and file_read_diff trying to buy

• Tool-Output Pruning — the same bottleneck attacked by prevention rather than compression, with the trade stated on both sides. A hard per-tool ceiling (500 lines, 100 matches, 10s) costs no model call, no extra prefill and no prefix-cache break — the three costs that made five of seven learned pruners inflate tokens somewhere in that page's grid, and the cache term still unpriced there — but it is entirely content-blind, so it truncates at the cap where a probe-driven head keeps the important ~30% wherever it sits. The two systems also disagree on where the boundary is: that page prunes between turns at the agent-environment boundary after the agent has read the full response once, while a cap means the full response never exists. Nobody has run cap-versus-compress on one harness

• Cost-per-Task Over Cost-per-Token — a rare instance where both terms move the same way at once, on a third-party harness with a fixed backend. Holding the model constant and changing only the review system takes tokens per sample from 5,664K to 385K (14.7×) and wall clock from 13m06s to 1m23s (9.5×) while SEM-F1 rises 2.17× — the harness-is-the-bigger-lever result that page draws from Writer's swap, reproduced in a different task domain by a party selling neither baseline. Two caveats keep it from being clean. Against Codex the token saving nearly vanishes (422K vs 525K) and the quality gap is larger, so the cost win is a property of the specific baseline's exploration behavior rather than of the constrained design. And no dollar figure appears anywhere — the third measured system in a row to price itself in tokens and seconds and never in money

• Review as the Control Point — the automated-reviewer-capability moderator, now with a frontier rather than a scalar. That theory treats reviewer capability as one construct; this says a review system occupies a region in precision–recall space that its architecture fixes and its backend only moves within, with the upper-right quadrant empty across all twelve configurations. Its report-everything-then-filter prescription is also implemented here in the strong form — the SubAgent decides breadth, a separate filter-only reflector decides what survives, and the filter is structurally unable to add findings — though with no ablation it is unknown what the split bought

• Repository Exploration Subagent — the delegate-versus-distribute contrast on the same bottleneck. FastContext pulls exploration out into one read-only subagent that returns compact citations so the solver's window stays clean; this pushes exploration down into one SubAgent per changed file, each with its own bounded read tools and its own window, with cross-file dependencies recovered on demand via file_read_diff. Both keep the exploration trail out of a single monolithic context and neither has been run against the other; the file-level partition is the paper's stated answer to the coherence-versus-efficiency trade-off, chosen over per-hunk or per-function granularity on the argument that finer partitions fragment a coherent change

• LLM-Judge Validation — this paper's every reported figure sits on one unvalidated semantic matcher, and its §6 defense ("all systems are evaluated by the same matcher under the same protocol, relative comparisons remain valid") is that page's thesis restated as a mitigation. No κ against human matching judgments, no order controls, no per-run variance from the five samples — and the five-sample mean is a homogeneous jury, which the measured ρ ≈ 0.66–0.97 says buys much less than independence predicts. The credit due: AACR-Bench's labels are expert-verified through three rounds by 80+ engineers, so the weak link is the matcher rather than the ground truth, which is the better of the two places to have one

• Verification as the New Bottleneck — automated review priced end to end for the first time in the corpus: 1m23s and 385K tokens per pull request for a pass that names 20% of the issues 80+ engineers found, against 13m06s and 5.7M tokens for 28.9%. Both numbers argue the bottleneck has not moved

• Claude Code, Codex — the two baselines, tested at v2.1.169 and v0.140.0 respectively, through each product's own review command

• Anthropic, OpenAI, Greptile — the vendors whose shipped review features the corpus's three review-efficacy measurements run through

• Reasoning–Acting Interleaving (ReAct) — the loop this system pins down. Its six capped tools, 30-iteration bound and empty-round detector are the enumerated-and-bounded end of the pattern CS329A lecture 4 introduces as an open-ended prompting abstraction

• Guarantees That Degrade at Deployment: Action-Space Soundness, Admissibility Without Effect, and a Vendor-Coupled Security Framework — this system's six capped tools filed as what does not recover the enumerated-action guarantee at scale: shrinking the action space until it fits is a design choice about what to build, not a recovery of soundness over a large space, and with no ablation of any kind the contribution of the bounded tool set is untested

Open Questions#

• The paper has no ablation of any kind, so which of the three deterministic injections carries the 2.17×, and whether any of it survives an instruction-only baseline, is untested — the same gap Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents leaves in its own domain. The falsifiable form is cheap and the code is open-sourced: run OpenCodeReview with reflection disabled, with rule-guided dispatch replaced by agent-driven triage, and with the six bounded tools swapped for a shell; then run Claude Code with OpenCodeReview's built-in rule text pasted into its prompt. Until at least the last of those exists, "determinism beats autonomy" is a design philosophy that has twice been measured against a confound rather than against its own alternative.
• Is 20% recall a ceiling of the constrained design or of the review task? Across all twelve configurations no system reaches both precision above 25% and recall above 25%, and within OpenCodeReview the best-precision backend is tied-worst on recall — so the empty upper-right quadrant of Figure 3 may be an architectural limit or may be the benchmark's. Distinguishable without new methods: sweep OpenCodeReview's reflector threshold and its 30-iteration bound to trace out its own precision–recall curve, and report whether the curve bends toward the empty quadrant or slides along the same frontier the baselines sit on.
• Every figure here is a Qwen3-235B-A22B-Instruct match verdict with no published agreement against human matching judgments, on a benchmark whose labels were human-validated to three rounds. Does the matcher agree with expert adjudication well enough to support 12-way ordering, and how much of the five-run mean's stability is genuine rather than the ρ ≈ 0.66–0.97 dependence a homogeneous jury carries? Both are answerable on a few hundred sampled match decisions plus the per-run spread the authors already have and did not print.

Sources#

• OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review — Zhengfeng Li, Lei Zhang, Xianwei Wu, Zhengqi Zhuang, Yingjie Xu, Boge Wang, Shaofei Zhu, Chuan Wang, Peng Zhao, Xinyu Zheng & Guoping Rong (Alibaba Group / Nanjing University / Peking University), OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review, arXiv 2608.09290, v1 2026-08-10 / v2 2026-08-11, empirical (11pp, 3 tables, 6 pictures; code at github.com/alibaba/open-code-review). §1 (the two weaknesses and three challenges; the 2.17× and 5–15× headline), §2.1–2.3 (the three prior lines and their locality bound; the cited MSR-2026 field study putting agent-only PRs at a 45.2% merge rate against 68.4% human-only with 60% of rejected agent-only PRs below a 30% signal-to-noise ratio), §3.2 (the four-tier rule chain, replace/merge modes, the six-stage file filter including the 80%-of-context-window cut), §3.3 (the ReAct loop, the 30-iteration bound, empty-round detection, the 60%/80% compression thresholds, the six bounded tools, the three-stage line-anchoring fallback), §3.4 (the asymmetric information boundary, falsification-not-verification, filter-only, fail-open on parse failure), §4.1–4.4 (AACR-Bench setup, the twelve configurations, the precision-not-recall-inflation reading, cost efficiency, the precision–recall regimes), §5 (determinism as a design principle; information boundaries vs model boundaries; the cost-quality frontier), §6 (threats — the five-run judge mean, the same-matcher defense, and the useful-comment-counted-as-false-positive concession). COI: seven of eleven authors are Alibaba Group, the system is Alibaba's, and both baselines are competitors' products; the benchmark (AACR-Bench, arXiv 2601.19494) shares four authors with this paper including the second and last, so the evaluation instrument is substantially in-house. Disclosed by author affiliation only — the paper states no competing-interest note.
• Per-table verdicts. All three tables were garbled by docling and hand-repaired at ingest; the raw now holds the repaired versions and captions sit above all three. Table 1 collapsed 4 rows → 1, Table 2 collapsed 6 → 1, and Table 3 was collapse and shift combined, with cells from different (model, system) rows interleaved across columns — the most damaged table in the corpus to date. Table 3 was reconstructed cell-by-cell from pdftotext -f 9 -layout. Verified at compile by three independent identities across all twelve rows, which a scrambled row mapping could not satisfy: (i) SEM-F1 reproduces the harmonic mean of the printed Precision and Recall in every row (max residual 0.09pp); (ii) Match/GT ÷ 1505 reproduces the printed Recall in every row (max residual 0.05pp); (iii) Match/Gen reproduces the printed Precision in every row (max residual 0.05pp). Table 1 reconciles against §3.2.2's prose (ad-hoc highest, built-in "base tier", first-match-wins) and Table 2 against §3.3.2's "fixed set of six" plus Figure 2's rendered tool list — three independent confirmations of the six tool names. Do not "correct" the small residuals: §6 states the rate metrics are the mean of five stochastic judge runs while the match counts are pooled, so a mean of ratios is being printed beside a ratio of means. Two defects belong to the paper, not the parse: §4.2's "4,580 across 200 PRs" contradicts Table 3's 435/5980 for the same configuration and only the table is consistent with the printed 7.23% precision; and §4.1.2's "six models, excluding GPT-5.5" describes the five Claude Code rows that Table 3 and Figure 3's twelve points actually contain.
• En-dash corruption caught and corrected at ingest: "5–15× fewer tokens" had rendered as "515 ×" in the Introduction and again in §4.3. Verified at compile — token ratios reconstructed from Table 3's five paired Claude Code rows run 5.4× (GLM-5.1) to 14.7× (Claude-4.6-Opus), so "5–15×" is the correct range; likewise §4.3's "1.3–2.2× higher SEM-F1" (measured 1.27–2.17×) and "2.5×" against Codex (21.00/8.36 = 2.51×) at "comparable token cost" (422K vs 525K, i.e. OpenCodeReview is 20% cheaper, a conservative statement).
• Figures opened per the image two-pass rule. Figure 3 (precision–recall scatter) axes read Precision (%) 0–40 and Recall (%) 0–30, both linear with no multiplier; its twelve points confirm Table 3's precision/recall pairs, and the empty upper-right quadrant plus the two Claude Code points above OpenCodeReview's maximum recall are read from the plot. Its caption's "at comparable or lower recall" is contradicted by its own plot and by Table 3; §4.4's prose is correct. Figure 2 (architecture) confirms the four rule tiers, the per-file SubAgent fan-out, the six-tool list, the line-match-then-LLM-relocation path, and the reflection block. Two mismatches with the prose worth recording: the figure gives each SubAgent an explicit "Planning" stage that §3.3.1 never describes, and labels the reflector's input "File Context" where §3.4.1 specifies the diff. Neither affects a quoted value. The remaining four images are the banner, licence and layout decorations.
• Ingest parse health: verify: ok on all 8 checks including table-collapse: 0, table-shift: 0, and canary-recall 12/12 (recall 1.00) after the table repairs — a genuine pass rather than a skipped or vacuous one, and no Stage preprocess failed anywhere in the docling run, so no page was silently dropped.
• Claude Code Changelog — Anthropic, Claude Code CHANGELOG (vendor-claim, rolling document snapshotted 2026-08-03, scoped v2.1.200–2.1.220; release notes only, no rationale attached to any entry). Cited here solely to date the paper's baseline: v2.1.202 ("Changed /review <pr> back to a fast single-pass review; use /code-review <level> <pr#> for the multi-agent review at a chosen effort level"), v2.1.206 ("Improved /code-review findings quality on claude-opus-4-8 across all effort levels"), v2.1.215 (Claude no longer self-invokes /verify and /code-review), v2.1.218 (/code-review moved to a background subagent). The snapshot starts at v2.1.200, so it says nothing about what v2.1.169's /code-review did — the claim supported is only that an explicitly multi-agent, effort-tiered /code-review had shipped at least 33 releases before this paper was submitted, which bounds §6's "current state-of-the-art in industry-shipped coding agents." Full treatment of the document on Optimizer–Evaluator Decoupling.
• How Bridgewater Built an AI Analyst That Does Hours of Expert Research in Minutes — McManus, Ran & Weight (Bridgewater Associates), LangChain channel, 2026-07-24, 25:44 talk, case-study. Cited for the third determinism-beats-autonomy instance: ~4x codegen latency vs Claude Code and near-instant re-execution, from three un-ablated injections (18:08–25:29). First-party and unmethodologized throughout — see Agentic Code Generation as Compilation for the evidence caveats on every figure