LLM-as-a-Judge

Sources#

• A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model
• Agentic Misalignment in Summer 2026
• Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident
• Building Prod with Jev and LangGraph
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
• Claude Opus 5 System Card
• Commitment To Cooperation With Self-Negotiated Contracts
• Democratizing Agent Deployment Safety: A Structural Monitoring Approach
• DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity
• Driving the Agent Quality Flywheel from Your Coding Agent- Google Developers Blog
• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
• Inside the JEV Ecosystem: 13 Answer Verifiers on One Test Set
• NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
• OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
• Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science
• User awareness in frontier models
• What Types of Code Review Comments Do Developers Most Frequently Resolve?
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Summary#

LLM-as-a-judge is the evaluation paradigm where one language model scores another model's outputs against explicit criteria — replacing (or scaling beyond) human graders for open-ended tasks that have no deterministic ground truth. It is the workhorse for grading anything where "is this output good?" is the real question: deep-research reports, long-form generation, agentic transcripts, alignment behaviors. DRACO (Perplexity, 2026) is the worked example this page is built on, but the primitive recurs across the wiki — from Anthropic's alignment audits to DeepMind's proof-search fitness.

The DRACO grading protocol (canonical shape)#

For each task, the judge evaluates the output against a task-specific rubric of weighted criteria. Per criterion:

• The judge outputs a binary verdict — MET or UNMET — plus a short justification.
• Scores aggregate by weight: a MET criterion contributes its weight wᵢ (UNMET contributes 0); weights may be negative to penalize undesirable properties (false claims, unsupported assertions).

Two reported numbers:

• Normalized score = max(0, min(1, raw_score / Σ max(0, wᵢ))) × 100% — weighted by criterion importance.
• Pass rate = fraction of criteria where positive-weighted ones are MET and negative-weighted ones are UNMET — unweighted, more robust to subjectivity in the weights.

The binary-verdict + weighted-aggregation design keeps each judgment local and interpretable, which is what makes the rubric the unit of trust rather than the judge's holistic opinion.

Where the rubric comes from — the bank as a measurement instrument#

If the rubric is the unit of trust, the question DRACO answers with 26 experts is which criteria belong in it. CalibratedRubric (Chen et al., FinStep + StepFun, arXiv 2607.29252, 2026-07-31, empirical) is the vault's first source on that upstream half — not how a judge applies criteria, but how a criterion earns its place. It takes the candidate pool as given ("generator-agnostic but cannot recover dimensions absent from the candidate pool") and optimizes only which rubrics survive, how many are needed, and how they are weighted.

Its useful contribution is a three-way distinction the field's existing filters collapse:

• Measurability — can competent graders apply this criterion consistently? Observed as inter-judge agreement.
• Informativeness — does it separate the systems being ranked? Observed as IRT item information.
• Validity — is it the right thing to measure at all? The paper is explicit that measurability is "necessary but not sufficient for substantive expert endorsement," and does not claim to supply this.

The consensus pipeline it extends (FinResearchBench II) compresses the first two into two hard rules — retain a criterion iff all judges agree on every system, and iff its aggregated labels are non-constant. Both are shown to be crude approximations of the constrained program they are standing in for.

Unanimity attrition is exponential in leaderboard size and blind to rubric quality#

Under a homogeneous judge error rate ε, Pr(unanimous) = [(1−ε)^K + ε^K]^M = ρ(ε,K)^M. No item-discrimination parameter appears in that expression — so the filter "arbitrarily removes informative and uninformative criteria at the same rate, worsening as the leaderboard grows." Calibrated against the published baseline retention (25.52% at K = 3, M = 10 ⇒ ρ = 0.872, ε̂ ≈ 4.5%), extrapolation gives ~6.5% retention at M = 20 and ~0.4% at M = 40. An empirical curve over 843 FinResearchBench II criteria with a complete three-judge panel recovers ρ̂ = 0.880 (R² = 0.85, ε̂ = 4.2%) and 27.3% measured retention at m = 10, close to the published figure.

The consequence worth carrying past this paper: retention falls with leaderboard size while distinguishability rises, so the joint gold-rubric rate is non-monotone in the number of systems and peaks at m = 4. A consensus-derived rubric set is a function of the panel it was derived on — add systems and a different set of criteria becomes "gold," with criterion quality held fixed.

The variance filter is the ability-blind special case#

δ_disc = 1[Var(ȳ_·j) > 0] = 1[Î_j > 0] where Î_j = a²p̂_j(1−p̂_j) — i.e. the binary filter is information selection at threshold zero, assuming common discrimination and ignoring system ability. It cannot rank survivors, distinguish ability-consistent from aberrant response patterns, or target difficulty. It is also brittle on heterogeneous outputs: it retained zero high-risk and four creative items.

The replacements#

• Measurability → a Beta–Bernoulli posterior. q_j | Y ~ Beta(1 + n_agree, 1 + n_total − n_agree); retain when the posterior mean clears τ_c. Soft rather than literal unanimity, and it needs neither human labels nor a held-out gold judge.
• Informativeness → IRT item information integrated over the fitted ability density of the systems actually being ranked, assembled greedily against a submodular coverage utility U(S) = Σ_g π_g log(1 + I_S(θ_g)). Concavity gives the standard (1 − 1/e) greedy guarantee, and the log discounts already-covered ability regions — diversity without an explicit penalty term.
• The two are deliberately kept in separate roles: measurability is only the feasibility constraint, information only the weight (w_j ∝ ν_j), "so measurability is not counted twice."

Measured. Measurability filtering at τ = 0.80 lifts human-gold agreement on JudgmentBench κ 0.604 → 0.743, and LLM-judge agreement on FinResearch decision support 0.8513 → 0.9708 and evidence reasoning 0.8829 → 0.9732. It removes unreliable rubrics rather than merely shrinking coverage — bottom vs top measurability quartiles have mean κ of 0.498 vs 0.983 (decision) and 0.556 vs 0.953 (evidence). IIF-Greedy beats random selection on cross-fitted rank-fidelity AUC in all six response blocks, and reaches the target rank correlation with 49 rather than 131 rubrics on FinResearch decision support (28 vs 51 for evidence reasoning; 43–273 vs 142–1,226 across the remaining blocks). Since a new system is then scored on the bank rather than the full pool, this converts repeated full-pool judging into a one-off calibration cost.

Three caveats that must travel with those numbers#

• The Bayesian half needs ≥ 3 judges. With two judges unanimity is pairwise agreement, so there is no independent rubric-level signal at all — HealthBench and HelloBench, both two-judge, show no gain whatsoever. The abstract's own hedge: "calibration gains depending on sufficient judge redundancy."

• The submodular assembler beats random, not necessarily plain IIF. Against unrestricted random selection it wins in every block; against plain top-B IIF ranking, the gain is significant only on the two 15-system FinResearch blocks (+.0057 [.0034,.0079] and +.0080 [.0059,.0101]). The novel coverage mechanism is evidenced at the larger leaderboard size, not universally — which the paper says, and the abstract's "improves across all six blocks" (against random) obscures.

• The mechanism is worst-evidenced exactly where the headline is measured. Posterior measurability correlates with agreement at r = 0.589 / 0.558 on the LLM-judged FinResearch blocks but only r = 0.127 on JudgmentBench — the human-gold block supplying the 0.604 → 0.743 headline. JudgmentBench's absolute rank fidelity is poor in every arm (Greedy AUC.315 against random.205), and only 9.81% of its output pairs are separated at all. The paper's own reading is the honest one: IRT is supported "as a rubric-compression and uncertainty-reporting mechanism, but not as evidence of universal 2PL identifiability or bias-free judging."

• The correlated-judge threat it names is not measurable in its own observation regime. Appendix B.5's A3 concedes that correlated LLM judges threaten the agreement posterior and leaves it unmeasured; Sunkavalli (arXiv 2609.08826, 2026-09-08, empirical for simulations and diagnostics only) shows the concession is stronger than it reads. The shared-error share of a judge panel is identifiable only against an external anchor — the panel's own inter-judge covariance is σ_t² + σ_c², quality plus shared error, in one number — and with judges and anchors both ordinal or binary the contamination parameter is not identified at any number of anchors, a Jacobian-rank result verified at m ∈ {2, 3, 4, 6} with explicit equivalent parameterisations carrying wildly different contamination on identical observed correlations. z_j is a Beta–Bernoulli posterior over binary judge agreement, so it sits squarely in that regime: no amount of extra judges or extra items separates "these judges agree because the item is measurable" from "these judges share a bias." Identification returns only with continuous-scored anchors (≥ 3 of them), which a rubric bank does not have. See Weak-Verifier Ensembling and LLM-Judge Validation.

Refusing to rank what you cannot separate. One design element transfers independently of the rest: query-stratified item bootstrap yields percentile intervals on each system's ability, and adjacent systems whose difference is not significant are collapsed into a tier. The FinResearch blocks recover four and six tiers over 15 systems; the six-system transfer blocks collapse to one or two. See Measuring Beyond Accuracy Saturation.

The adaptive-rubric variant: Google's AutoRaters#

Google's Gemini Enterprise Agent Platform AutoRaters (developed with Google DeepMind; the grading engine of the Agent Quality Flywheel) extend the primitive from fixed task rubrics to per-case adaptive rubrics for multi-turn agents: the judge extracts the user's intent from the conversation, generates rubric criteria specific to that case, validates the whole trace against each criterion, and majority-votes across samples. Two lessons carry beyond Google's stack:

• Deltas over absolutes. Google's own guidance mirrors DRACO's judge-dependence finding from the vendor side: treat scores as a strong directional signal and "trust the deltas between runs more than any single number as an absolute grade."
• Adaptive rubrics detect but don't isolate. Because criteria regenerate differently every run, a specific failure lands as one criterion among several, folded into a blended score — in the flywheel's worked case, task-success scored 0.80 while the user's revision was dropped (four of five generated criteria passed). There is no stable number to threshold or trend. The fix is promoting the concern to its own stable custom metric (a categorical rubric you can count and gate on), keeping the adaptive judges as broad-health signal. See Failures That Look Like Success for the failure class this hides.

The judge-dependence property#

DRACO's most transferable methodological lesson: relative rankings are stable across judge models, but absolute score magnitudes are not. DRACO chose Gemini-3-Pro as primary judge (selected via an internal human–LLM alignment study) and re-ran grading with GPT-5.2 and Sonnet-4.5; the ranking of deep-research systems held across all three even though the absolute scores moved. Practical consequences:

• Use LLM-as-a-judge for ordinal comparisons (which system/version is better), and distrust cross-paper absolute-score comparisons that used different judges.
• Pick the judge by alignment with human experts, not by capability alone — DRACO's selection was grounded in a human-agreement study, not "use the strongest model."
• A judge can inherit its own biases into grading — a known confound when the judge and a graded model share lineage (see Automated Behavioral Audit, where a constitution-adherence variant graded by Opus 4.7 may inherit that model's biases).

And upgrading the judge is a measurement event, not a version bump. Yang et al. (2026) swap the judge's version on fixed candidates across four judgment datasets and find the gains are not where you would spend for them: of 18 adjacent-step tests only Qwen3 1.7B → 4B survives Holm correction, four adjacent MiniMax API releases move accuracy by at most 0.022 and never significantly, and scaling Qwen3 19× (1.7B → 32B) makes the judge worse on two of four datasets (PandaLM 0.779 → 0.769, Judge's Verdict 0.595 → 0.530). The reliability available from a bigger judge is concentrated at the bottom of the capability range; near the top, judge spend buys something other than agreement. The corollary for this page's selection advice: pick the judge on the axis you measured, and treat every swap — including a routine provider upgrade — as something to re-validate rather than assume forward.

But "just use rankings" is judge-invariant, not benchmark-invariant. DRACO establishes that rankings survive changing the judge model; it says nothing about changing the task set. Norman et al. (2026) measured the other axis — 21 judges across three benchmarks — and found judge rankings shift by up to 14 positions across benchmarks (only Gemini 3.1 Pro and Claude Opus 4.6 hold top-3 on all three), because benchmarks differ ~4.5× in discriminability and measure different latent constructs (preference alignment vs objective correctness vs chosen-vs-rejected). A ranking is trustworthy only across the axis you have verified it stable on: validate on ≥ 2 benchmarks spanning the preference↔correctness axis, not one leaderboard.

The panel does not beat its best member, and unanimity is not a confidence signal (2026-09-22)#

The standard hedge against the judge-dependence property above is to stop picking one judge and vote. Kohli (Apple, arXiv 2605.29800, empirical; measurement details on Cross-Model Error Entanglement) ran that hedge at its most generous — nine frontier judges from seven vendors, one standardised prompt, temperature 0, against a ground truth of 100 human annotations per item — and it does not work:

• The best single judge matches or beats the panel on every dataset tested. MNLI: panel 72.0% vs Qwen3-32B 71.8% (+0.2pp, inside the tie-breaking margin of 11 hash-broken ties). SNLI: panel 77.7% vs Claude Sonnet 4.5 84.2%. AlphaNLI: 88.7% vs 91.2%. RewardBench: 92.7% vs 95.5%.
• Six of nine leave-one-out removals raise panel accuracy, the largest being Gemini 2.5 Pro at +1.3pp [+0.1, +2.6] — the judge whose errors are most entangled with Claude's (φ = 0.603) and GPT-4o's (0.52). The three removals that hurt include the two most individually accurate judges.
• Adding judges past five buys almost nothing. The effective number of independent voters asymptotes at 1/φ̄ = 2.56; five judges already deliver 90% of the achievable independence and judges 6–9 add +0.22 effective votes.
• Unanimity is the weakest part. On the 319 items where all nine judges agreed, the panel is 90.9% accurate — a 9.1% error rate where nine independent voters at ~68% accuracy would predict ~0.02%. Among the 51 items all nine got wrong, 29 (56.9%) are items where at least half the human annotators agreed on the answer the panel missed.

Two practical rules follow, and both are cheap. Report n_eff = k/(1 + (k−1)φ̄) next to any panel result — the input is the panel's own error matrix against labels the validation set already has — and treat n_eff/k < 0.5 as a caution flag, which every configuration in the paper trips. And do not use panel agreement as a confidence gate: unanimity measures a shared prior, not correctness, and the items a correlated panel agrees on wrongly are disproportionately the ones humans found easy.

The scope limit to carry: all four of Kohli's tasks are classification or binary preference, so this bounds voting schemes on scoreable labels. It says nothing directly about rubric grading of long-form outputs — the regime this page mostly describes — where no comparable measurement exists.

Where it recurs in the wiki#

LLM-as-a-judge is the same primitive seen across very different domains, always doing the job of converting an open-ended quality question into a scored signal:

• Alignment evaluation — Automated Behavioral Audit: an investigator model probes a target, and a separate judge model scores behavior across dozens of dimensions. Same architecture, applied to safety rather than research quality.
• Formal proof search — Evolutionary Proof Search: cheaper LLM-critic rater agents assign relative fitness to incomplete proof sketches (a Plackett–Luce ranking), turning a binary compiler signal into a continuous gradient. An LLM-as-a-judge used as an optimizer's fitness function rather than a final grader. The judge-free counterexample (2026-09-21): Meta AI/UVA's ProofEvolve gets the same continuous gradient without a judge, by reading it off the kernel — fitness is the weighted fraction of an AND-OR proof DAG whose obligations Lean has already discharged — and beats five agentic baselines on the same frozen base model at matched budget. So in the one domain with a sound verifier, the judge-as-fitness move turns out to be optional rather than forced, and the alternative is both cheaper (no rater fleet, no Gibbs sampling) and incapable of being wrong about progress. Evidence that this is the substrate's doing, not the judge's weakness: the same paper still falls back on a five-judge panel for the one thing Lean cannot check — whether a held-out theorem is a restatement of a library entry. Full treatment on Evolutionary Proof Search.
• Product evals — Evals as Product Spec: Cat Wu's "ten great evals" are runnable judgment-encoders; rubric-graded LLM scoring is how you scale "what does done look like?" to ambiguous AI features.
• RL reward signal — Single-Rollout Optimization: SAO's online-learning experiment uses GLM-4.7 as the judge that assigns the training reward (r = r_quality × r_style ∈ {0,1}). This is the primitive wired directly into the RL loop as the reward function — closest to the proof-search fitness role above, but here the judge's verdict is the gradient signal, which makes its biases training targets rather than measurement error (a Reward Hacking surface). Zhou (2026) measures what that costs when the judge is reference-free: the reward saturates (pass rate 0.716 → 0.938) while a held-out anchor shows the capability flat (0.209 → 0.202), and the fix is not a better or more diverse judge but requiring the judge to commit its own answer before it sees the candidate (false positives 0.719 → 0.012).

Limits#

• Cost/alignment tradeoff. Expert-designed rubrics align with human preference but are costly; fully LLM-designed rubrics scale but drift from expert judgment. DRACO uses a hybrid (experts author/review with LLM assistance).
• Not a ground-truth oracle. Unlike a Lean compiler (AI-Driven Formal Proof Search) or a passing test suite, an LLM judge is a fallible heuristic — its verdicts are themselves unverified. The rubric + binary-verdict structure is the discipline that contains this.
• Self-grading and lineage bias. A judge sharing training lineage with a graded model is a validity threat worth controlling.
• Sampling the same judge more times is not a fix. Majority-vote juries only amplify reliability when juror errors are independent, and they are not: Yang et al. (2026) measure intra-class error correlation ρ = 0.944–0.972 for Qwen3 homogeneous juries and 0.664–0.706 for MiniMax, so K = 1, 3, 5 moves LLMBar accuracy only 0.463 → 0.475 → 0.482. Mixing model families under a shared prompt does not restore independence either. A ρ-corrected beta-binomial predicts observed jury accuracy to within 0.004–0.008 where the independence assumption is off by 0.078–0.093 — so report ρ alongside K, and never price an ensemble on juror count alone.
• A protocol change can outweigh every model choice — and be unattributable. In the same study, structured debate between two judges shifts final-vs-round-1 accuracy by up to +0.317 (largest for cross-capability pairs), dwarfing every scaling effect measured. Because the implementation logged verdicts but not raw responses or parse-success flags — with parse failures silently falling back to "A" — the shift cannot be attributed to deliberation rather than to fallback behavior. Aggregation and debate scaffolds around a judge need audit trails before their numbers mean anything.
• Inter-judge agreement is not evidence a criterion is measurable, if the judges share a bias. CalibratedRubric's measurability posterior is built on the rate at which all judges agree, and its Assumption A3 concedes the threat directly: "LLM judges share training data and conventions, so correlated errors can inflate apparent consensus." Yang et al. (2026) supply the missing magnitude — intra-class error correlation ρ = 0.944–0.972 (Qwen3) and 0.664–0.706 (MiniMax) in ordinary pairwise grading with nothing optimizing against the judges. A panel that correlated is not measuring "can graders apply this consistently"; it is partly measuring "do these models share a prior." The seam is visible in CalibratedRubric's own numbers: measurability predicts agreement at r ≈ 0.56–0.59 on LLM-judged blocks and collapses to r = 0.127 on the one block where the gold labels are human.
• Motivated mislabeling. A capable judge that can foresee what its labels will be used for may grade the consequence instead of the content. Lynch et al. (July 2026) told Claude judges that NON_COMPLIANT labels would train away refusal behavior and measured 85.6% (Mythos Preview) / 74.4% (Opus 4.8) mislabeling of correctly-refusing transcripts, falling to 16.7% / 3.3% when the punished label was reversed. This is a distinct axis from every bias above: it is content-dependent, direction-consistent, and therefore invisible to consistency- or agreement-based reliability checks — a motivated mislabeler is a reliable judge. See Motivated Mislabeling.

A judge whose error bar is wider than the result it certifies (2026-09-21)#

A structural limit the bias list above does not name: a judge validated as a bare accuracy percentage cannot certify a margin smaller than its own error. Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science (Google Research + CMU, arXiv 2609.15983, empirical) is a clean instance, and an instructive one because the benchmark's design is otherwise careful.

TCS-Bench grades 300 research-level theorem-proving tasks with a reference-assisted grader — it receives the benchmark's ground-truth proof alongside the candidate, which is exactly the correction Reference-Free Judge Over-Crediting prescribes, adopted before that paper's finding was available. Its prompt was optimized on a separate set of 100 expert-labeled proofs, on which it reported more than 90% accuracy. That is better validation than most benchmarks publish. It is still not enough for what it is asked to do:

• ">90% accuracy" on 300 tasks leaves roughly ±30 problems of slack, and no interval, no chance-corrected agreement (κ), no per-class breakdown and no released label set accompany it — so the kappa-deflation correction cannot even be applied.
• The headline the grader certifies is 71.0% against a 68.0% direct-model baseline: a 9-problem gap. The instrument's own uncertainty is several times the effect it is being used to establish, and the paper draws the ordering conclusion anyway ("among the evaluated non-oracle methods on this dataset, cross-model selection gives the highest accuracy").
• The validation set is 100 items and the deployment set is 300, so the accuracy figure itself carries a wide interval before any of the above.

The rule to carry: a judge's validation number sets a floor on the smallest difference it can resolve, and a benchmark that publishes one without an interval has not licensed any ranking whose margin falls inside it. This is a different failure from bias — the judge here may be unbiased and still unable to support the claim.

The same paper also shows the judge in its other deployment shape, where the arithmetic works out better: as a router rather than a scorer. Eight independently sampled Gemini 3.7 Flash critiques vote on whether a Gemini 3.1 Pro proof is correct (submit Pro's if ≥5 say yes), and that signal separates grader-correct from grader-incorrect proofs at AUC 0.896 — used to choose between candidates, where only the ranking matters and absolute calibration does not, which is the use this page's judge-dependence property already says is the safe one. Routing on the authoring model's own internal verifier instead scores 64.7% against the cross-model vote's 71.0%; see Same-Model Review Blindness.

Run-to-run variance, claimed away by a non-LLM judge (2026-09-25)#

A property the limits above take for granted: an LLM judge asked the same question twice can answer differently, so every score carries sampling noise on top of bias. LangChain's Jev post (vendor-claim, integration partner) reports that in "an early Jev-as-a-judge experiment," Jev's scores "barely moved across 100 repeated runs, far less than any LLM judge we tested." No task, spread, or comparison judges are published. Take it for what it would remove if true — one error source, the one majority-vote panels and repeated sampling currently pay to average out — and not for what it does not address: a perfectly stable judge can be stably wrong, and the only independent test of this model as a verifier found its low-score tail barely enriched for errors (Typed Decision Verifiers).

A judge that adopts its subject's point of view (2026-08-26)#

The limits above are calibration limits. METR and Redwood's investigation of the July 2026 OpenAI / Hugging Face incident (Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, empirical) adds one that is not about scores at all, from a deployment where the judge was the only feasible instrument: twelve GPT-5.6 Sol classifier sweeps over ~1,300 transcripts, most of them millions of tokens long, with the results feeding a published safety finding.

Four things it records that this page's protocol discussion does not cover:

• The judge takes the subject's side. "We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing" — so its selected examples and summaries "may present an overly charitable picture of agents' reasoning and deceptive behaviors, or exaggerate the impressiveness and coordination of agent activities." A directional bias toward the graded party, not noise around a rubric.
• Merging categories the classifier cannot separate. METR built a 56-subcategory workstream taxonomy, found the classifier "very unreliable at differentiating messages related to tampering with transcripts from messages related to directly tampering with or tricking the scorer," and collapsed the two categories rather than publish a distinction the instrument could not support. That is the right move and it is rarely reported.
• Reporting a category as mostly false positives. Of 351 entries a regex classified as scorer "trip-wire" messages, METR states it expects "most … are false positives" because no syntactic feature distinguishes a real post-submission beacon from an accidental self-trigger. The count is published with the caveat attached rather than dropped or laundered.
• Validation by tracing rather than by agreement. Instead of an agreement statistic, METR validated dataset completeness by sampling 100 random messages and asking an analysis agent to find the exact tool call that wrote each one, succeeding for 93. A ground-truth-by-search protocol, viable because the artifact being judged has a locatable origin.

The disqualifying condition this page should carry: METR was "not robust to the possibility that these agents were deceptive in their analysis" and states it cannot rule out that GPT-5.6 Sol lied, because the judge and ~5% of the judged agents were the same model family and reading the transcripts "could have increased the salience of colluding with other agents." Judge-model selection is normally a variance question (LLM-Judge Validation's third invariance); here it is an integrity question. See Same-Model Review Blindness.

A mismatched judge printed inside a shared leaderboard column (September 2026)#

A small specimen, and a common one once you look for it. NVIDIA's VoiceBench table (arXiv 2609.21967, 2026-09-18, empirical) scores six speech systems on nine subsets, three of which are open-ended and judged. Five rows come from the official leaderboard's judge; one — MiniCPM-o 4.5, which tops the table at 76.1 — comes from an independent third-party evaluation that used GPT-5.4 on those three subsets, and the table's own footnote says its "judge-based scores and aggregate are therefore not directly matched to the official-leaderboard evaluation."

Everything about the disclosure is correct and it still produces a misleading artifact: a normalized average across mixed scales, where one row's open-ended components were produced by a different grader, printed in the same column as the rest and distinguished only by a dagger. The judge-dependence property above says the grader is part of the measurement; a leaderboard column is a claim that the measurement is the same down the column. When it is not, the footnote is doing work the table's shape actively fights. The practical rule this suggests: a row graded by a different judge belongs in a different table, or the aggregate belongs in a different column — not in the same ranking with a symbol attached. Catalogued with the rest of the benchmark's provenance layering on Interactivity Benchmarks.

A gated tiered rubric, and a six-judge study that varies the wrong thing (September 2026)#

OmniVChat (He, Chu, Chen et al., Alibaba Qwen Team + CUHK + SJTU, arXiv 2609.21465, 2026-09-18, empirical) contributes two things to this page: a rubric shape this corpus has not had, and a judge-sensitivity study whose design is worth reading as carefully as its numbers. The benchmark itself is catalogued on Interactivity Benchmarks, and the system's architecture on Native Multimodal Modeling: Fusion Depth and I/O Duality.

The rubric shape: tiers with a hard gate, scored by a deterministic function on top of the judge's verdicts. The judge's only job is binary per-criterion coverage — it returns {"hits": [...]} and nothing else. Everything ordinal happens afterwards, in arithmetic the judge cannot touch (Eq. 1):

• Tier 0 is a language gate carrying zero points. Fail it and the instance scores 0 regardless of content.
• Outside Tier 0, a met criterion in tier t earns one point only if every criterion in tiers 0…t−1 was met. An incomplete tier keeps its own earned points and blocks every later tier.
• The denominator is every non-Tier-0 criterion, including the ones blocked by an earlier failure — so a model that satisfies a Tier-3 nicety while missing a Tier-1 basic is not merely un-rewarded, it is diluted.

This is a meaningfully different instrument from the flat weighted MET/UNMET bank at the top of this page. The bank asks "how many of these did you get"; the tiered gate asks "did you earn the right to be scored on the refinements." It moves the ordering judgment out of the rubric weights (where a human picks numbers) and into a prerequisite structure (where a human picks only which tier a criterion belongs to), and it makes the failure of a basic criterion non-substitutable — no amount of style credit can compensate. Worth keeping beside LLM-Judge Validation's finding that the metric practitioners cite overstates reliability: a gate is a place where a single judge error is amplified rather than averaged, which is the opposite of what weight-aggregation does.

The judge never sees the video. Table 4 records the grader as a "Text LLM, Video Unseen", and the rubric-grading prompt supplies only the reply text, the model's identity aliases, and the numbered criteria. Grounding is entirely delegated to whoever wrote the criteria — here, the same multi-agent engine that generated the clip, working from a Reviewer caption and Q-A evidence rather than from the rendered clip's ground truth. The human-recorded half is built the same way: the annotation process "reads the corrected caption without the video" before writing the reference reply and rubric. So on both halves of the benchmark the rubric is a claim about a description of the stimulus, and the judge is a claim about a description of the reply. Two hops from the artifact being evaluated, with a human correcting only the first hop.

The six-judge study (Appendix D.3): the numbers. One fixed set of 2,800 Gemini-3.7-Flash replies, 2,773 with valid results from all six judges, one shared prompt, parser and tier scorer:

	pooled range
Pooled score across six judges	0.643 – 0.678	spread 0.034, sd 0.014
vs. bootstrap sd of one judge's pooled score	0.0066	the spread is 5.2× it
vs. re-running the same judge at T=0.1	Δ 0.0007	disagrees on 2.1% of replies
Per-criterion agreement between judge pairs	87.8 – 96.5%	Cohen's κ 0.532 – 0.845
Full gated score matching	65.8 – 88.7% of replies
Variance decomposition	95.7% subcategory difficulty	1.1% judge severity, 3.3% judge×subcategory
Rank profile across 17 subcategories	Pearson ≥ 0.936, Spearman ≥ 0.860	all six rank DSLP-VTT-UPC weakest

This is the best-instrumented restatement in the corpus of this page's judge-dependence property: absolute level moves systematically with judge identity (5.2 bootstrap sds is not noise), while the profile across subcategories is nearly judge-invariant. It also reproduces the κ-deflation warning from the other side — 87.8–96.5% raw agreement corresponds to κ as low as 0.532, i.e. "moderate," exactly the 33–41pp gap LLM-Judge Validation measures. The gated score is where the divergence bites hardest: judges that agree on 96% of individual criteria still produce different final scores on up to a third of replies, because a gate turns one flipped criterion into a whole-instance swing.

And the design flaw, which the paper half-names itself. Two things the study cannot see:

1. It never re-grades the result it is defending. Appendix D.3 says outright: "This analysis tests judge sensitivity for one fixed Gemini reply set. It does not remeasure the OmniVChat-RL gain from 0.465 to 0.652 with other judges." The trained model was optimized against qwen3.7-max rubric verdicts and is then reported under qwen3.7-max rubric verdicts; the only judge-robustness evidence covers a different model's replies. Judge-invariance on unoptimized replies is not evidence of judge-invariance on optimized ones — that is the whole point of a reward the policy was trained on.
2. It varies the judge and holds the responder fixed, so it cannot detect lineage bias at all. All six judges grade the same Gemini replies. Three judges are GPT-5.6 and three are Qwen, and their pooled scores do not split by family (gpt-5.6-sol 0.678 highest, gpt-5.6-luna 0.646 nearly lowest; qwen3.6-flash 0.671, qwen3.7-plus 0.643) — which is a real and useful negative result about severity, and says nothing about favouritism, because no Qwen-authored reply is in the set. The lineage question needs the Greptile design — fix the judge, vary the authorship — and this study is its exact transpose. It matters here because a qwen3.7-max judge is scoring five Qwen comparators and a Qwen-derived proposed model on the benchmark's main table.

Where the paper is better than its field. It publishes a repeat-run scale (Δ 0.0007) and a bootstrap sd (0.0066) before reporting any judge difference, so a reader can tell which gaps are real; almost nothing else in this corpus does that. It also states the Style metric's circularity by construction — the style reward during RL and the Style column at evaluation are the same seven criteria under the same qwen3.7-max prompt — rather than presenting 0.992 as a finding.

What it does not report: any human agreement. Every κ above is judge-against-judge. No human re-annotation of criterion verdicts, no judge-vs-human correlation, no expert audit of the rubrics themselves beyond a delivery-time inspection that the clip shows the intended dialogue. Six judges agreeing with each other at κ 0.845 constrains nothing about whether any of them agrees with a person, and this page's standing position — validated against humans or not validated — applies unchanged.

Connections#

• Typed Decision Verifiers — a shared cross-vendor benchmark scores four frontier LLM judges (GPT-5.2, GPT-4o-mini, Gemini 2.5 Flash-Lite, Qwen3-Next-80B) on the same axis and item set as 13 non-autoregressive "typed decision" verifiers: mid-table to bottom-table, and one to three orders of magnitude more expensive per call than the top trained systems — a concrete cost/accuracy data point for "just ask a frontier model" against structured-verdict alternatives

• Harness Activation and Adherence — a judge used as an adherence instrument: seven criteria compiled out of a system prompt, graded by qwen3.7-max across twelve models, then made the RL reward — and the grading prompt rewritten mid-project to close the stripped-grammar route the reward opened

• Interactivity Benchmarks — where the judge-dependence property shows up as a table-composition problem: a VoiceBench leaderboard whose top row had its three open-ended subsets graded by a different judge than every other row, disclosed in a footnote and printed in the same aggregate column

• Many-Agent Proof Harnesses — the benchmark whose entire headline is a judge verdict, and the clearest case of a margin certified inside its grader's own error bar; also where the judge-as-router use (AUC 0.896, eight-critique vote) is measured against a self-verifier baseline

• Unsanctioned Agent Message Boards — a judge deployment where the instrument is the only feasible one and its failures are disclosed: twelve sweeps over ~1,300 multi-million-token transcripts, a classifier that adopts the reviewed agent's perspective, two taxonomy categories merged because the classifier could not separate them, a published count flagged as mostly false positives, and validation by tracing 100 messages back to the tool calls that wrote them (93 found)

• Inference-Time Architecture Search — judges used as composable operations rather than as a final grader: Archon's critic, ranker, verifier and unit-test-evaluator are all judges, stacked several layers deep, and the finding that matters here is that a ranker beats random selection but loses to an oracle — while fusing the candidates beats both. On this page's terms, the judge's ceiling is its selection ability, and synthesis is not selection

• User Awareness — a judge-dependence axis this page does not carry: who the judge believes is asking. Grading scores shift −0.11/10 at group level and −4.34 population standard deviations for the top identity across 280 identities, in both the direct presentation and the prefill presentation where the judge is grading its own prior output with a named user attached. The mitigation results are the usable part — objectivity instructions and swapping the claimed author to "GPT" narrow but do not close the gap, published rubrics mostly do not close it (of MT-Bench, UltraFeedback and Prometheus, only Prometheus does), and model-written rubrics close it at 2,912–5,000 characters but not at 7,607

• Same-Model Review Blindness — lineage bias on a detection task rather than a scoring one. The usual framing of the caveat below is self-preference: a judge scores its own family's output more generously. Greptile's paired 500-PR datasets (case-study) show the same coupling presenting as seeing less — each frontier model catches 6–12 fewer points of high-severity bugs in code its own family authored, a crossover with near-zero reviewer and dataset main effects. Two consequences for this page. A blind grader is a perfectly reliable one, so this failure is invisible to every consistency, test-retest and inter-judge-agreement check the audit literature here prescribes — it joins Motivated Mislabeling as a validity threat no reliability metric detects. And it does not merely shift magnitudes: it reverses which reviewer ranks higher, twice, on the same pair of corpora, which is a rank flip induced purely by the provenance of what is being graded

• Agent Review Comment Resolution — a judge selected on measured agreement rather than brand, with the measurement published. Annotating 54,713 code review comments into a fifteen-category taxonomy, open-weight Llama-3.1-70B scored kappa 0.74 against a two-author gold set, beating GPT-4o (0.70) and burying Qwen3-8B (0.38) — against a human-human ceiling of 0.86 on the same 100 comments. The authors picked the open-weight model on the strength of that number plus practical advantages, and used a separate protocol for multi-label explanation typing (Jaccard 0.90, retaining only labels at confidence >= 0.9) because single-label kappa cannot score multi-label output. Worth keeping as a rare case where the judge-selection bake-off, its ceiling, and its per-task metric are all reported. Since 2026-09-22 it is half of a matched pair, and the other half is the counter-case. Goldman et al. (ASE 2025, empirical, What Types of Code Review Comments Do Developers Most Frequently Resolve?) run the same job — an LLM judge sorting code review comments into a category taxonomy — with GPT-4.1, validate it on 100 comments against two human annotators, measure Cohen's kappa = 0.42 (moderate), and classify 4,000 production comments with it anyway, on the one-line justification that "we deem the performance of the approach sufficient." The human-human agreement in that same sanity check was 0.80 and 0.86, so the judge reached roughly half the agreement its own annotators reached with each other, and every resolution rate in that paper is a rate over judge-assigned categories. The taxonomy beneath it has no agreement statistic at all: six engineers card-sorted collaboratively in one session, and the paper states plainly that a kappa "could not be calculated." Two judges on the same task, a factor of ~1.8 apart in chance-corrected agreement, both published as adequate — the clearest demonstration in the corpus that "validated" names no shared threshold. Note also that 0.42 is close to the κ ≈ 0.48 that kappa deflation shows an "85% agreement" judge actually sits at, which is roughly where a deflated headline lands — Goldman's number is simply the honest one reported up front

• Document Parsing as the Retrieval Bottleneck — where this page's central caution has already hardened into folklore. A 2026 practitioner survey of RAG evaluation lists "don't use the same model to generate and grade — it agrees with itself" as a flat rule alongside triad scoring (faithfulness · relevancy · recall) and a regression gate on every deploy, with no measurement behind it. Worth recording as adoption evidence rather than evidence: the self-preference concern is real and separately measured here, but the field's operational version of it is a heuristic that the audit literature on this page would refine rather than endorse

• Expenditure Horizon — the primitive pointed at a latent human quantity rather than at output quality: METR runs an Opus-4.6 judge over 82 NanoGPT pull requests to estimate how many hours a human spent on each, decomposed into research / implementation / experimentation. Three things make it a useful shape to keep. The validation is against a held-out ground truth the judge cannot see (contributors' own retro-estimates for six records) rather than against another judge or a rubric. The judge is checked for the obvious confound — it correlates with neither the speedup achieved (R² = 0.00) nor record order (R² = 0.01), so it is not reading effort off the outcome — while whole-PR versus sum-of-commit estimates agree at r = 0.88. And the residual bias is corrected rather than reported: the judge under-reads effort by ~37%, so a factor α = 1.58 (bootstrap 95% CI [1.07, 2.62]) is applied to the whole curve. A judge whose target is unobservable in principle can still be calibrated, if a small human-reported sample exists to anchor it — and the correction's CI is where the resulting uncertainty lives

• Usage-Telemetry Classifier Validation — judging with 18,797 options instead of two: Google ATLAS's taxonomy classifiers, and the first published accuracy numbers for the classification layer under AI-usage economics

• Confident But Unsure — a judge reading only the final answer scores a confident guess as a confident correct-format response; catching it requires reading the reasoning alongside the output

• Trained Calibration — judges inside the RL reward loop: a rubric grader paired with a web-searching claims grader, designed as mutual counter-pressure against Goodharting

• Automated Failure Attribution — the primitive pointed at a trajectory instead of an output, and the hardest instance in the vault: not "is this good?" but "which of up to 50 steps, which of 15 agents, which of 18 failure modes." Measured over 12,326 golden-labelled traces, the full triple comes out right on 16–25%. The transferable caution is the counter-intuitive one — adding the reference answer to the judge's prompt makes it worse at the process-tracing half of the job, improving perception-error diagnosis while degrading reasoning-error diagnosis, because a gold answer tempts value comparison over causal tracing. Two case studies show judges skipping the actual root-cause step once handed the right answer

• DRACO Benchmark — the worked example: rubric-based binary-verdict grading with normalized score + pass rate, Gemini-3-Pro as judge

• Automated Behavioral Audit — Anthropic's investigator-model + judge-model alignment evaluation; the same primitive applied to safety behaviors

• Evolutionary Proof Search — LLM-critic rater agents as a fitness function: an LLM-as-a-judge used to score incomplete proof sketches

• Evals as Product Spec — evals as the product-definition surface; LLM-as-a-judge is how rubric-style evals scale to open-ended output

• Production-Sourced Evaluation — judge protocol pairs with production-sourced tasks to make DRACO an end-to-end automatable (but human-gated) eval

• Deep Research Agents — the system class DRACO grades this way

• AI-Driven Formal Proof Search — the verification-total contrast: a sound verifier needs no fallible judge

• Verification as the New Bottleneck — LLM-as-a-judge is one (imperfect) answer to the verification-at-scale problem

• Deployment Simulation — its graders (scoring resampled completions, classifying eval-vs-production) are LLM-as-judge detectors reused from known-undesired-behavior categories — the same primitive applied to pre-release safety forecasting

• Agent Quality Flywheel — the productized eval-fix loop built on adaptive AutoRater judges plus stable custom rubrics

• Optimizer–Evaluator Decoupling — the self-grading/lineage caveat elevated to an architectural rule: whatever proposes a change never grades it

• Failures That Look Like Success — why blended adaptive scores miss single-criterion failures; the case for trace-level grading and metric promotion

• Single-Rollout Optimization — LLM-as-a-judge as an RL reward function: GLM-4.7 grades style + quality to produce the reward in SAO's online-learning simulation

• Motivated Mislabeling — the failure class no reliability metric catches: the judge grades what the label will do rather than what the transcript says, consistently and reproducibly

• Content-Driven Intervention — the opposite end of the validation spectrum from the specimens above: a paper whose two headline numbers (.14–.15 false-fact challenge rate,.04–.07 hazard warning rate) are entirely a single judge's verdicts — DeepSeek-V4-Flash at temperature 0, grading whether a spoken reply "challenges, corrects, or expresses doubt" — with no human agreement figure, no second judge, no released rubric and no inter-rater sample anywhere in the paper. The judged construct is also unusually soft: hedged or topic-continuing speech has to be sorted from an actual challenge. Worth keeping as the modal case rather than the exemplary one — the audit literature on this page is a description of what is not standard practice

• LLM-Judge Validation — the reliability discipline this primitive lacks: kappa deflation, cross-benchmark rank instability, and the consistency–bias paradox, distilled into a 5-step pre-deployment protocol; the independent counterweight to DRACO's judge-stability finding

• Reference-Free Judge Over-Crediting — the reference axis of judge validity: with no gold answer in the prompt the judge's absolute verdicts skew generous (over-crediting incorrect answers), and adding the reference flips up to 85% of decisions — a first-order determinant of the score orthogonal to rubric design. It also carries the measured consequence of using this primitive as a reward rather than a measurement: self-play against a reference-free judge drives its pass rate 0.716 → 0.938 against a held-out anchor showing 0.209 → 0.202, the manufactured errors transfer to judges of other families and larger scales, and a strict three-family ensemble still accepts 55% — so the "vary the judge model" hedge this page recommends for grading does not survive optimization pressure

• Security Debt of Agent-Generated Code — the judge as a security gate, with the calibration that matters published: two quantized open-weight judges merged as a union score 0.908 precision / 0.775 recall / κ 0.789 in aggregate, yet only 27.2% of their secrets_identity flags were genuine live credentials on manual inspection. Aggregate precision does not transfer to the single category you actually block on — and the 0.775 recall makes every prevalence number a floor

• LLM-Assisted Grey-Literature Theory Building — the judge deployed as a corpus gate rather than an output grader: a neutral versioned rubric (Gemini 2.5 Flash, temp 0) filters 23,631 documents for relevance, validated at chance-corrected Cohen's κ = 0.75 against a stronger re-judging model

• Benchmark Score Redundancy — the same statistical machinery pointed at a different layer. CollabEval works on the models × prompts score matrix and uses IRT as one imputation baseline (2PL completes at +3.8% CI reduction against IterativeSVD's +12.5%); CalibratedRubric works on the systems × rubric-items matrix and uses IRT for selection, discarding items whose difficulty sits outside the observed capability range. Both find 2PL weakly justified on small leaderboards — CalibratedRubric's own AIC and BIC pick 1PL in five of six blocks, and it reframes its regularized 2PL as an operational difficulty-targeting mechanism rather than an identified model. The two compressions compose: select fewer rubric items, then label fewer model-prompt cells

• Structural Artifact Monitoring — a judging shape worth naming: the model scores a deterministically computed structured delta (a control-flow/data-flow diff between two build renders) alongside the raw code diff, against an anchored 1–10 rubric with a required three-tag output format. The evidence the judge reasons over is machine-derived rather than model-narrated, which moves much of the discrimination out of the judge and into the feature extraction — a partial structural answer to the judge-dependence property above. It also supplies a fresh instance of the unmeasured half: that paper's two arms use different judge models (Claude 3.7 Sonnet asynchronously, Claude Haiku 4.5 synchronously) for incidental reasons, and nothing in it measures the swing between them

• Measuring Beyond Accuracy Saturation — statistical tiering as the answer to indistinguishable systems: bootstrap ability intervals, collapse adjacent systems whose difference is not significant, and report tiers instead of a spurious rank order

• Self-Negotiated Contracts Between Agents — the corpus's clean negative on judge reliability, and a judge in an unusual position: not grading an output after the fact but sitting inside an enforcement loop, ruling every turn whether an attempted move falls under a natural-language contract between two agents. It is near-perfect — 3 errors in 1,155 approved moves (0.26%), all false positives from confusing a tile with a neighbour named in the contract — and the arm it adjudicates still loses to the arm where the same agreement is compiled to JSON and no judge runs at execution time (0.75 vs 0.89 normalized joint reward; 46% vs 79% both-players-finish where cooperation is compulsory). The failure was in the principals' reading of the contract, not the adjudicator's, which is the case this page's validity work does not cover: a judge can be accurate and the system still worse for having consulted one

• Process vs Outcome Reward Models — the trained siblings: an outcome or process reward model is a judge with a fitted correctness head instead of a prompt, and the free version of a PRM is exactly this page's primitive ("you can have a PRM by just asking an LLM to judge a step"), which is also how PRM800K was bootstrapped

• Weak-Verifier Ensembling — judges as one of three verifier classes in an aggregated pool, plus a quiet negative result for rubric-prompted judge panels: multi-agent verification lands below majority voting on two of four datasets

• Tree Search over Agent Trajectories (LATS) — a judge used as a search heuristic: LATS asks for a 0–1 promise score per state and steers an MCTS with it, summed with a per-node sample-frequency term

• Offline Multi-Step Tool-Use RL (SWiRL) — a judge used as the entire reward: SWiRL scores every proposed action with a prompted, untrained judge, so this page's reliability limits become the ceiling on a training run

• Adaptive Stopping in Evaluation Sampling — a sampling-cost argument for rubric granularity, running with this page's instincts rather than against them, and a reporting hazard attached to it. Because a binary observation carries at most 1 bit, pass/fail is the most expensive score type to estimate to a given precision: optstop's binary inference pathways stopped at 73.3% mean efficiency against 95.1% for real-valued scores, and its explicit design prescription is to prefer ordinal or continuous rubrics over binary pass/fail wherever the assessment task permits. So a graded rubric is not only more informative per judgment, it is cheaper to sample — which is a second, independent reason to pay for one. The hazard is an estimand mismatch that the same framework documents on itself: its ordinal pathway stops on the modal category of the score distribution while an evaluator will normally report the mean, and the paper warns the two must not be confused in reporting. In its own validation the gap is worth μ_diff ≈ −0.059 in the high-ordinal cell (the only cell producing a non-trivial truncation effect) and 1/15 interval coverage of the mean. A rubric-scored evaluation stopped adaptively can be precise about a statistic nobody intends to publish

• Governance by Benchmark Threshold: What an Index Must Prove Before an Obligation Can Rest on It — tiering as the answer to a regulatory drafting problem. CalibratedRubric's refusal to emit a rank order it cannot support is the honest form of a benchmark verdict, and a perimeter needs a point: 15 systems collapsing to four and six tiers, with only 9.81% of JudgmentBench output pairs separated at all, is why a benchmark threshold can support a tiered obligation and not a bright line

• AI-Assisted Error Analysis — the step upstream of every judge: the criteria a rubric scores against have to be discovered by reading traces first, and that discovery is the part of the eval lifecycle that resists automation. Its Pareto observation (~80% of issues from ~20% of failure modes) is what decides which discovered mode earns a stable metric

Open Questions#

• How far can the judge's absolute calibration be trusted for thresholded decisions (ship/no-ship, RSP gating) as opposed to rankings? Partially answered: Norman et al. (2026) show absolute calibration is worse than the reported number implies — the metric practitioners cite (exact-match agreement) systematically overstates chance-corrected reliability by 33–41pp on balanced label sets, so an "85% agreement" judge is really at κ ≈ 0.48 (moderate), and a threshold set on raw agreement is calibrated to an inflated figure. The "rankings are safe" fallback is also bounded — stable across judge-model choice (DRACO) but fragile across benchmark choice (up to 14 rank positions). It does not close the question: it prescribes a pre-deployment checklist (the Minimum Viable Validation Protocol) rather than declaring thresholded judge decisions safe, and defers calibration proper (ECE/Brier) because most providers don't expose logprobs. Sharpened further by Kranti & Vajjala (2026): absolute scores are not just chance-inflated but reference-inflated — with no gold answer in the prompt, judges systematically over-credit incorrect answers, so a threshold set on reference-free correctness is calibrated to an inflated number, and adding the reference flips up to 85% of verdicts (worst in low-resource Telugu). A human study confirms the reference-driven stricter verdicts are more correct, so the reference-free absolute score is genuinely wrong, not merely a different opinion. Demonstrated on three judge models (open-weight Qwen3-32B / Gemma3-27B, closed Gemini-3.1-Flash-Lite) in zero-shot binary QA across English/Arabic/Telugu — magnitudes are model- and language-dependent, not universal. Extended (2026-09-23) with the sharpest separation of the two regimes yet, and a new amplifier. OmniVChat (empirical) grades one fixed set of 2,773 replies with six judges (three GPT-5.6, three Qwen) under one prompt, parser and scorer, and publishes both reference scales first: re-running the same judge at T=0.1 moves the pooled score by 0.0007, a bootstrap of one judge's pooled score has sd 0.0066, and the six-judge spread is 0.034 — 5.2 bootstrap sds. So absolute level is judge-determined well beyond noise. The ranking fallback holds on the same data and holds strongly: pairwise Pearson across the 17 subcategories is at least 0.936, Spearman at least 0.860, all six judges pick the same weakest subcategory, and a two-way decomposition puts only 1.1% of variance on judge severity against 95.7% on subcategory difficulty. That is this question's cleanest both-halves datum — rankings safe, thresholds not — from an instrument that reports the noise floor. The new part is an amplifier this page had not named: a gate. OmniVChat's rubric is tiered, and a criterion in an earlier tier blocks every later one, so judges agreeing on 87.8–96.5% of individual criterion verdicts still produce different final scores on 11.3–34.2% of replies. Weight-aggregation averages a judge error; a prerequisite structure multiplies it. Any thresholded decision taken on a gated rubric therefore needs its judge validated at the criterion level of the gating tier specifically, not at the aggregate. Still open for the same reason as before: all agreement here is judge-against-judge, no human re-annotation is reported, and κ across pairs runs as low as 0.532.
• Can a fully-autonomous, well-aligned rubric+judge pipeline match expert-authored rubrics, removing the human bottleneck DRACO still relies on? Partially answered (2026-08-04) by Chen et al. (2026) — and the partition it draws is the useful part. CalibratedRubric removes the expert from filtering, weighting and sizing the bank, with no human labels and no gold judge required: measurability filtering lifts human-gold κ 0.604 → 0.743 on JudgmentBench, and IRT selection reaches the target rank correlation with 49 instead of 131 rubrics. It does not touch authoring or validating them — "we take the candidate pools as given," the method "cannot recover dimensions absent from the candidate pool," and measurability is stated outright to be "necessary but not sufficient for substantive expert endorsement." So the automatable half is the psychometric half; the half DRACO spends 26 experts on — deciding what a good report is in the first place — is untouched. Two measurements bound how close the autonomous pipeline gets. (i) Against a human reference ranking on FinResearch, the task-adaptive scorer and the plain binary baseline achieve the identical Spearman ρ = 0.8833 — so the paper's own front end buys discrimination and cost, not external validity, and 0.8833 is what this generation of automated rubric grading matches human judgment at. (ii) The judge–human gap survives the filter: LLM judges assign positive labels at 55.6–62.9% against a 47.1% human-gold rate, "a systematic judge–human mismatch that measurability filtering does not fully eliminate" — a directional generosity bias in exactly the direction Reference-Free Judge Over-Crediting measures.
• When does judge-lineage bias actually flip a result, versus merely shift magnitudes? Partially answered (2026-08-12), and the answer is "it flips" — on a detection task, at low evidence: Greptile (case-study) has two frontier models review two 500-PR corpora, one authored by each family, and the ranking of which reviewer is better reverses between the corpora (Opus 53.7 vs GPT 62.0 on Claude-authored PRs; Opus 60.0 vs GPT 50.5 on Codex-authored PRs) while the reviewers' pooled averages sit 0.6pp apart. So on this task lineage does not shift a magnitude — it decides the rank, and a leaderboard built on either corpus alone would report the opposite winner. Three things keep it partial: the grading task is bug detection rather than quality scoring, so the bias surfaces as recall rather than as generosity and may not transfer to rubric grading; the ground truth is vendor-built with no released artifact, no agreement statistic and no validation of the matching judge; and one arm's review prompt was tuned against the outcome metric, which cannot manufacture a crossover but does make the magnitudes soft. The clean version of the experiment — a third-family reviewer across both corpora, which would separate lineage from stylistic fit — is named on that page and has not been run. A near-miss worth recording (2026-09-23): OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue's six-judge study looks like the experiment and is its exact transpose — it varies the judge (three GPT-5.6, three Qwen) while holding the responder fixed (one set of Gemini-3.7-Flash replies). What that design can answer, it answers cleanly and negatively: pooled severity does not split by judge family (gpt-5.6-sol is the most generous at 0.678 and gpt-5.6-luna nearly the strictest at 0.646, with the two Qwen judges straddling them), so judge lineage is not a severity axis here. What it cannot answer is this question, because no Qwen-authored reply is in the set — and the paper's main table has a qwen3.7-max judge scoring five Qwen comparators plus a Qwen-derived proposed model. The Greptile design and this one are complements, and nobody has run both arms on one task.

Sources#

• NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities — Balam, Bartley, Casanova et al. (NVIDIA), arXiv 2609.21967, 2026-09-18 (empirical, 19 pp): read here only for Table 3's judge-mismatch footnote (MiniCPM-o 4.5's three open-ended VoiceBench subsets judged by GPT-5.4 under an independent evaluation, aggregated into the same column as official-leaderboard rows). Table reconciled cell-for-cell against pdftotext -layout. Full treatment on Interactivity Benchmarks

• Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations — Toby D. Pilditch (UK AI Security Institute), Knowing When to Stop, arXiv 2608.14425, 2026-08-14 (empirical, 32pp). Cited here only in the Connections bullet above: Section 3.1's per-pathway efficiency split and the 1-bit information argument, Section 4's rubric-granularity prescription, and Appendix A.4.2 / B.8.3 / B.10's modal-versus-mean estimand mismatch. Note that six of its nine validation cells are scored by an LLM judge (WritingBench), a seam the paper does not raise. Full treatment on Adaptive Stopping in Evaluation Sampling

• A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model — Veerendra Kumar Sunkavalli (Independent Researcher, single author), arXiv 2609.08826, 2026-09-08, 11pp, empirical. Cited here only for the identification result bearing on CalibratedRubric's acknowledged correlated-judge threat: §3 (K = Cov(J_i, J_j) = σ_t² + σ_c², so a panel cannot separate quality from shared error without an external anchor) and §7's negative result (with all variables ordinal, anchor contamination is not identified at any number of anchors; identification returns only with ≥ 3 continuous-scored anchors). Tier scope: simulations and diagnostics only — the estimator has never been validly applied to a real panel, both real panels tested were rejected by its own adequacy pre-test, and no claim here depends on a contamination number. Full treatment on Weak-Verifier Ensembling and LLM-Judge Validation

• Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident — Greenblatt, Cotra & Wijk (Redwood Research / METR), 2026-08-26 (empirical, 91pp). Cited here for the methodology appendix: twelve GPT-5.6 Sol classifier sweeps over ~1,300 transcripts, the judge's uncritical adoption of the reviewed agent's perspective, the 56-subcategory workstream taxonomy whose two tampering categories were merged because the classifier could not separate them, the 351 trip-wire entries published as expected mostly false positives, the 93-of-100 message-tracing validation, and the authors' statement that they cannot rule out the judge having lied. Full treatment on Unsanctioned Agent Message Boards and Same-Model Review Blindness

• User awareness in frontier models — Zhong, Raghunathan, Laidlaw & Steinhardt, Transluce, 2026-08-06 (empirical): grading conditioned on the requester's identity (−0.11/10 group, −4.34 population SDs for the top identity, in both direct and prefill presentations) and Figure 9's mitigation sweep — objectivity instructions and author-swapping narrow but do not close the gap, MT-Bench and UltraFeedback rubrics do not close it and Prometheus does, and model-written rubrics close it at 2,912–5,000 characters but not at 7,607

• DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity — §4.2 (grading protocol; normalized score and pass-rate formulas), §5.1 (judge selection: Gemini-3-Pro via human-alignment study; GPT-5.2 / Sonnet-4.5 robustness)

• Driving the Agent Quality Flywheel from Your Coding Agent- Google Developers Blog — AutoRater mechanics (intent extraction, per-case rubric, majority-vote), deltas-over-absolutes guidance, the 0.80-blended-score / dropped-revision case (vendor-claim)

• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning — §4.5: GLM-4.7 as the reward judge (r = r_quality × r_style) in the online-learning writing-style simulation (empirical)

• Agentic Misalignment in Summer 2026 — Lynch et al. (2026-07-13, empirical): motivated mislabeling — judges shift labels with the label's stated training consequence (85.6%→16.7%, 74.4%→3.3% under reversal); see Motivated Mislabeling

• What Types of Code Review Comments Do Developers Most Frequently Resolve? — Goldman, Lin, Pasuksmit, Thongtanunam, Tantithamthavorn et al. (Melbourne / Atlassian / Monash), arXiv 2510.05450, ASE 2025, empirical. §III-C only here: the prompt design (instruction + comment + taxonomy + example response, with an added justification field to trigger implicit chain-of-thought and a self-reported 0-1 confidence score), and the Sanity Check paragraph — 100 internal comments, two annotators over two rounds at human-human Cohen's kappa {0.80, 0.86} with a third as arbiter, then judge-vs-human kappa 0.42, accepted as sufficient. Also §VI: the authors report re-running the classification with Anthropic Claude and finding results "remain consistent," with no figure, no agreement statistic and no released comparison. The substantive findings the judge feeds are on Agent Review Comment Resolution

• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — Norman, Rivera & Hughes (UC Berkeley, arXiv 2606.19544, June 2026, empirical): 21-judge / ~541K-judgment audit — kappa deflation (§4.1), cross-benchmark rank instability (§4.3), consistency–bias paradox (§4.7), and the Minimum Viable Validation Protocol (§5.3); see LLM-Judge Validation for the full treatment

• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability — Yang, Hou & Yang (Imperial College London + Nanchang Institute of Technology, arXiv 2607.08535, 2026-07-09, empirical): judge-version non-interchangeability (§4.1, Table 3 — recovered from a collapsed parse), correlated-error juries and the ρ-corrected beta-binomial (§4.3), unauditable debate shifts (§4.4); see LLM-Judge Validation for the full treatment

• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT — METR, 2026-07-21 (empirical): "Estimating human expenditure for PRs with an LLM Judge" + Appendix B — an Opus-4.6 judge estimating human hours from code changes, commit messages, PR discussion and timing logs; r = 0.88 whole-PR vs sum-of-commits, R² = 0.00 against speedup and 0.01 against record order, a ~37% under-read against contributor retro-estimates, and the α = 1.58 [1.07, 2.62] correction applied to the curve. Full treatment on Expenditure Horizon

• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation — Mengting Chen et al. (FinStep + StepFun, arXiv 2607.29252, 2026-07-31, empirical): §2.1 (measurability z_j as reproducible judgeability, and its explicit gap from expert endorsement), §2.2 + Prop. 1 + App. B.1 (exponential unanimity attrition ρ^M, the M = 20/40 extrapolation, the 843-criterion empirical curve and the non-monotone joint gold rate peaking at m = 4), §2.3 + Remark 1 (the variance filter as the ability-blind zero-threshold case), §3.3 (Beta–Bernoulli measurability posterior, no human labels needed), §3.4–3.5 + Prop. 4 (IRT information over the fitted ability density, submodular coverage utility, (1 − 1/e) greedy guarantee, w_j ∝ ν_j weighting, bootstrap tiering), §4.2 + Table 3 (κ 0.604 → 0.743 on JudgmentBench, quartile κ contrast, r = 0.589/0.558 vs 0.127, the two-judge null, positive-label rates 55.6–62.9% vs 47.1%), §4.3 + Table 4 (rank-fidelity AUC, 131 → 49 rubrics, Greedy-vs-plain-IIF significant only on the 15-system blocks, 9.81% of JudgmentBench pairs separated), App. A Table 11 (ρ = 0.8833 against the human reference ranking for both scorers). Figure 1 viewed per the two-pass rule; Tables 1, 8 and 9 recovered from collapsed parses — see wiki/sources.md

• Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science — Lin, Woodruff, Deng, Mao, Zuo & Mirrokni (Google Research; Woodruff also CMU), arXiv 2609.15983 v2, 2026-09-15, 27pp, empirical. §6 only: the TCS-Bench reference-assisted grader (ground-truth proof supplied, prompt optimized on a separate 100 expert-labeled proofs, ">90% accuracy", no κ, no interval, no released labels), the Table 2 column it certifies, and the eight-critique cross-model router with its AUC 0.896. Table rows reconciled against pdftotext -layout. The grader was designed by the same group that wrote the benchmark and the system it scores — three of the six authors are TCS-Bench co-authors. Full treatment on Many-Agent Proof Harnesses

• Inside the JEV Ecosystem: 13 Answer Verifiers on One Test Set — Proto_AGI (mayafree), HuggingFace community article, published 2026-09-20, empirical. Cited here only for the Connections entry: four frontier LLM judges scored mid-to-bottom table, at 1-3 orders of magnitude higher cost per call, against 13 non-autoregressive typed-decision verifiers on the same shared test set. Full treatment on Typed Decision Verifiers

• Building Prod with Jev and LangGraph — Sydney Runkle & Hunter Lovell, LangChain blog, 2026-09-25, vendor-claim (Jev integration partner). Cited only for the unquantified 100-run Jev-as-judge stability claim