LLM-Judge Validation

Sources#

• An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
• Sidekick's continual learning loop
• User awareness in frontier models
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Summary#

Reliability is not validity. A judge can be perfectly reproducible — return the same verdict run after run — and still be systematically wrong: chance-inflated, benchmark-fragile, or deterministically biased toward one answer position. Norman, Rivera & Hughes (UC Berkeley, arXiv 2606.19544, June 2026) ran the largest systematic LLM-as-a-judge evaluation to date — 21 judges from nine providers, three benchmarks (MT-Bench, JudgeBench, RewardBench), three protocols (agreement, consistency, bias-audit), 118 runs, ~541,000 individual judgments, all at temperature 0 over a five-week March–April 2026 window — and found that the way judges are validated in practice (headline exact-match agreement) systematically overstates how good they are. The paper's contribution is not a new judge but a validation discipline, packaged as a five-step Minimum Viable Validation Protocol.

This page is the vault's independent counterweight to DRACO's reassuring "rankings are judge-stable" finding: the two papers measure different invariances, and together they bound how far a judge score can be trusted.

Finding 1 — Kappa deflation (the headline metric lies)#

Practitioners report a judge's exact-match agreement with human labels ("85% agreement!"). That number does not correct for agreement expected by chance. Cohen's κ (and Krippendorff's α) does. The gap — kappa deflation Δκ = EM − κ — is large and universal:

• On MT-Bench, every one of the 21 judges shows Δκ ∈ [33.8, 41.3] pp, cohort mean 38.6 pp. Even the best chance-corrected judge, Gemini 3.1 Pro, posts EM = 0.849 but κ = 0.511 — a 33.8 pp gap. A judge reporting "85% agreement" on MT-Bench has κ ≈ 0.48 — moderate, not the near-perfect band the percentage suggests.
• The deflation tracks the benchmark's label distribution, not the judge's quality. Balanced ternary MT-Bench (A/B/Tie, chance ≈ 1/3) → 38.6 pp mean; JudgeBench (pairwise correctness) → 23.7 pp; binary chosen-vs-rejected RewardBench → 10.2 pp. Balanced labels raise expected-by-chance agreement, which inflates the raw–corrected gap exactly as Cohen's correction predicts. The deflation is a property of metric × benchmark, so exact-match figures used to justify deployment "overstate discriminative ability by an amount that depends on the benchmark, not the judge."

Fix: report κ or α as the headline reliability number, with exact match demoted to a secondary figure.

Finding 2 — Single-benchmark validation doesn't transfer#

Judge rankings are not portable across benchmarks. Validating a judge on one leaderboard tells you little about its standing on another:

• 11 of 21 judges shift ≥ 4 rank positions across the three benchmarks; the abstract's headline is a shift of up to 14 positions. The extreme case is Llama 3.3 70B: MT-Bench #5 → JudgeBench #20 (a collapse); the opposite direction is Minimax M2.7: MT #17 → JB #5 (a jump). Only Gemini 3.1 Pro and Claude Opus 4.6 hold a top-3 position on all three benchmarks.
• Two coupled drivers. First, benchmarks differ wildly in discriminability: MT-Bench compresses all 21 judges into a 13.5 pp κ band (0.376–0.511, ~0.6 pp between adjacent ranks), while JudgeBench spreads the same judges over 60.4 pp (0.271–0.875) — 4.5× wider. Where the band is compressed, tiny κ differences produce huge rank swings. This is the MT-Bench ceiling effect: its preference-style label set can't separate strong judges. Second, the three benchmarks measure different latent constructs — preference alignment (MT-Bench), objective correctness (JudgeBench), chosen-vs-rejected discrimination (RewardBench) — and a judge strong on one can collapse on another.

Fix: validate on ≥ 2 benchmarks spanning the preference-style ↔ correctness-style axis, not the discriminability of any single dataset.

The DRACO reconciliation (two different invariances)#

The vault's prior answer on judge trust came from DRACO: rankings are stable across judge models, absolute magnitudes vary → use ordinal comparisons, distrust cross-paper absolute scores. This paper does not contradict that — it measures a different axis of variation:

• DRACO fixes the tasks + rubric and varies the judge model → the ranking of systems-under-test holds (Gemini-3-Pro, GPT-5.2, Sonnet-4.5 agree on the order). Judge-model-invariant.
• This paper fixes the judge protocol and varies the benchmark → the ranking of judges themselves is fragile (up to 14 positions). Benchmark-variant.

So the practical rule sharpens: a ranking is trustworthy only across the axis you have actually verified it stable on. DRACO earns "use rankings" for judge choice; it says nothing about benchmark choice, which this paper shows is where rankings break. The reassurance and the warning are the same lesson seen from two sides.

A third axis — varying the judge's version rather than its identity or its benchmark — arrives below with Yang et al. (2026).

Finding 3 — The consistency–bias paradox (reliability masking invalidity)#

The paper's sharpest diagnostic. High test-retest reliability (> 0.95) coexists with severe position bias (> 0.10) in two production-deployed judges:

Judge	Test-retest	Position bias	JudgeBench κ
Qwen 3 8B	0.992 (highest in cohort)	0.192 (highest)	0.289 (3rd lowest)
Gemini 2.5 Flash	0.988	0.125	0.578

The mechanism: test-retest measures the stability of a judge's outputs, not the correctness of its decision process. Position bias and within-judge agreement are mathematically orthogonal — a judge that deterministically favors whichever answer sits in position A achieves near-perfect test-retest (it's perfectly reproducible) while exhibiting maximum-possible position bias. The most reproducible judges can be among the least valid. Because reporting test-retest alone remains common practice, current validation "misleads precisely in the cases that matter most for deployment: highly reproducible judges."

This is reliability without validity compressed into a single failure mode — and the whole page's thesis in one number.

Finding 4 — Verbosity bias has largely faded#

A rare "this got better" result. All 21 judges register verbosity bias < 0.011 on MT-Bench (largest: GPT-4o-mini 0.010; 17 of 21 below 0.005) — an order of magnitude below the 20–40% length effects reported in 2023-era studies. Two model generations appear to have wrung most length-preference out of pairwise judging. Scope caveat (the authors are emphatic): this holds under a single pairwise rubric and one length-differential operationalization; it is not a claim that verbosity bias is solved under arbitrary rubrics or scoring tasks.

(Scope narrowed 2026-08-04 by Yang et al. (2026), empirical, and the caveat above is exactly where it lands. Under an active probe — appending a fixed, generic, content-free padding string to one candidate and measuring verdict change — verbosity bias on adversarial LLMBar runs 0.547 for Qwen3-1.7B and ≈0.13 for the MiniMax releases: one to two orders of magnitude above the < 0.011 measured here. The two numbers are not in conflict once the operationalizations are separated: Norman measures a passive length differential between naturally-occurring responses on MT-Bench, Yang measures induced sensitivity to padding on a benchmark built to be adversarial, over a partly-open-weight cohort that reaches down to 1.7B. Neither result generalizes to the other's setup. What survives is the narrower claim — verbosity bias has faded as a property of frontier judges on natural preference data, not as a property of judges — and the practical consequence is that a padding probe belongs in the bias-audit step alongside A/B reversal, because it is the thing that detects what the passive measurement misses.)

The third invariance — vary the judge's version (Yang et al., 2026)#

Yang, Hou & Yang (Imperial College London + Nanchang Institute of Technology, arXiv 2607.08535, July 2026, empirical) name the axis this page and DRACO both leave open. Call it evaluator-replacement ambiguity: when a score moves after you swap the judge for a newer or bigger one, the accuracy number alone cannot say why. The new judge may be more capable, differently biased, failing on a different slice, or simply parsed differently by the harness. Their reframing is the one this page argues for, stated at its most general — an LLM-judge result is a measurement, and replacing the instrument is a measurement-validity event, not a version bump.

Setup. Eight judges on two axes chosen because both are decisions practitioners actually face: a parameter axis (Qwen3 dense 1.7B / 4B / 14B / 32B) and a released-API axis (MiniMax M2 / M2.1 / M2.5 / M2.7, evaluated as shipped — explicitly not a controlled ablation, and the paper makes no causal claim about MiniMax internals). GLM-5.1 and mimo-v2-pro serve as cross-family reference judges outside both axes. Four datasets: LLMBar (419 adversarial pairwise), PandaLM testset-v1 (894 valid after dropping tie-majority items), a seed-42 2,000-example Chatbot Arena sample (1,997 valid), and Judge's Verdict (200 TechQA-derived, three-level pointwise). Near-greedy T = 0.1; exact two-sided McNemar on parse-shared examples (so significance reflects paired verdict changes, not differing parser coverage); Holm correction over the family of 18 adjacent tests.

Finding A — upgrading the judge is not a reliability intervention#

Of 18 adjacent-step tests, one survives Holm correction: Qwen3 1.7B → 4B, on LLMBar and Arena. All nine MiniMax adjacent tests fail to reach even uncorrected p < 0.05, with a maximum adjacent accuracy difference of 0.022.

Table 3 (reconciled against the PDF — see the parse note in wiki/sources.md):

Dataset	Best judge	Qwen3 1.7B → 32B
LLMBar	GLM-5.1, 0.900 [0.868, 0.925]	0.463 [0.416, 0.511] → 0.678 [0.632, 0.721]
PandaLM	MiniMax-M2.7, 0.857 [0.833, 0.878]	0.779 [0.751, 0.805] → 0.769 [0.740, 0.795]
Arena	mimo-v2-pro, 0.742 [0.722, 0.761]	0.625 [0.604, 0.646] → 0.688 [0.667, 0.708]
Judge's Verdict	GLM-5.1, 0.680 [0.612, 0.741]	0.595 [0.526, 0.661] → 0.530 [0.461, 0.598]

Read the two bold cells: 19× the parameters makes the judge worse on two of the four datasets — PandaLM 0.779 → 0.769, Judge's Verdict 0.595 → 0.530. Scaling the judge is not merely a weak intervention, it is not monotone. "Use the strongest model as judge" fails here on its own axis, not just against this page's cheaper-judge caveat.

The headline needs one correction the abstract doesn't make. "MiniMax adjacent releases show no gain" reads as a negative finding about MiniMax. Figure 2's per-model LLMBar accuracies show it is a ceiling effect on the panel, not a failure to improve: Qwen3 runs 0.463 / 0.617 / 0.647 / 0.678 while MiniMax runs 0.829 / 0.839 / 0.832 / 0.832, with GLM-5.1 at 0.900 and mimo-v2-pro at 0.883. The MiniMax series starts above where the entire Qwen3 sweep ends. So the honest form of the result is not "releases don't improve judges" but the reliability gain available from upgrading a judge is concentrated at the bottom of the capability range — the single robust step in the study is also its cheapest (1.7B → 4B, +0.154 on LLMBar) — and adjacent steps near the top buy nothing measurable on these datasets. That is a directly actionable procurement finding: past a threshold, judge-model spend buys you something other than agreement.

No judge wins all four datasets. GLM-5.1 leads LLMBar and Judge's Verdict, MiniMax-M2.7 leads PandaLM, mimo-v2-pro leads Arena. This is the same shape as Finding 2 above with a different thing varied — there, judge identity across benchmarks; here, judge version across benchmarks — and it converges on the same prescription: slice-specific measurement validity, not one-dimensional evaluator ordering.

Finding B — the capability–fairness association is strong and still insufficient#

On LLMBar the position-flip rate falls 0.320 (Qwen3-1.7B) → 0.117–0.147 (MiniMax releases), and across the eight evaluated judges LLMBar accuracy and position-flip rate correlate at Pearson r = −0.957. Granularity sensitivity shrinks too. But MiniMax-M2.7 — the best PandaLM judge in the panel — still changes 14.7% of its verdicts under pure A/B reversal. The authors are careful to call this an association measured on one dataset and one model panel, not a mechanism. Position randomization and slice-level bias reporting survive the upgrade. (The flip-rate metric here is not the same statistic as this page's |P(A wins) − 0.5| position-bias figure, so the numbers are directionally comparable but not on the same scale.)

Finding C — juries are capped by error correlation, and the cap is now a formula#

The jury experiment is a clean negative result with a usable correction attached. For homogeneous juries (one judge sampled K times at T = 0.7, deliberately not T = 0.1, since repeated low-temperature calls would understate juror dependence), the paper estimates the intra-class error correlation ρ from the vote matrix and compares three quantities — the independence prediction, a ρ-corrected beta-binomial prediction (q ~ Beta(α, β), α = ps, β = (1−p)s, s = 1/ρ − 1), and observed accuracy:

• Independence misses observed jury accuracy by median absolute error 0.078 (LLMBar) / 0.093 (PandaLM). The ρ-corrected prediction cuts that to 0.008 / 0.004, max error below 0.02.
• Measured ρ is high: Qwen3 homogeneous juries 0.944–0.972, MiniMax lower but still strongly dependent at 0.664–0.706.
• So jury size barely moves anything: Qwen3-1.7B on LLMBar at K = 1, 3, 5 gives 0.463, 0.475, 0.482.
• Heterogeneous juries also underperform Poisson-binomial independence predictions — mixing model families under a shared prompt does not restore independent errors.

The design question shifts from "how many jurors?" to "how independent are their errors?", and the reporting rule is ρ alongside K, never K alone.

This is the measurement sitting under Zhou's proof. Zhou showed analytically that no monotone aggregation rule can reject a region all judges accept when they threshold a shared latent signal, and measured pairwise acceptance correlation φ = 0.29–0.38 between three judge families — but under deliberate optimization pressure, on manufactured errors. Yang measures the same dependence with nothing optimizing against the judges at all — ordinary pairwise grading, ρ = 0.66–0.97 — and supplies the closed form that converts it into a prediction rather than a warning. Two independent routes to the same conclusion: more judges is not a reliability amplifier, and the family-mixing hedge fails on both.

Finding D — the largest protocol effect in the study is the one that cannot be attributed#

Structured debate (two judges, up to three rebuttal rounds, LLMBar) produces final-vs-round-1 accuracy shifts that dwarf every scaling effect in the paper: Qwen3-1.7B paired with GLM-5.1 / MiniMax-M2.7 / mimo-v2-pro shifts +0.317 / +0.305 / +0.289; Qwen3-4B with GLM-5.1 shifts +0.243 to reach 0.897. Same-family and top-vs-top pairs shift only +0.005 to +0.050, so the effect tracks the capability gap between the paired judges.

And it is uninterpretable. The implementation logged round verdicts and final verdicts but not raw responses or parse-success flags; round-1 parse failures fall back to "A" and later failures retain the previous verdict. Fallback rates are therefore unrecoverable without a full rerun, and the shift cannot be separated from "the weak judge's unparseable outputs got overwritten by the strong judge's verdicts." The authors present it as an auditability case study, not a deliberation result — which is the honest call, and the reason this finding earns a place on a validation page: a protocol intervention larger than any capability intervention in the study, whose mechanism is unrecoverable because six lines of logging were missing.

What this adds to the MVVP#

The MVVP is a pre-deployment checklist for one judge. Yang's Table 4 is a reporting standard for a judge result, and three of its six items are things the MVVP does not ask for:

Reliability risk	Required report item
Dataset-slice shift	Valid N, parseable N, tie handling
Parser artifacts	Parser success rate, fallback rule
Position/verbosity bias	A/B reversal, padding probe
Correlated jury errors	ρ and K, not K alone
Debate protocol artifacts	Raw outputs, parser status, round verdicts
Statistical overclaiming	Paired tests, Holm correction, uncertainty intervals

The parser/slice items (rows 1–2) and the ρ item (row 4) have no MVVP counterpart; row 3 is MVVP step 2 plus the padding probe; row 6 is the statistical hygiene the MVVP leaves implicit. The two documents are complementary rather than overlapping — validate the judge before deploying it (MVVP), then report what the pipeline around it actually did (audit trail).

Robustness checks (which the paper keeps deliberately narrow): human-ceiling calibration shows PandaLM retains headroom — best judge matched leave-one-annotator-out κ = 0.753 against a human ceiling of 0.920 — while Judge's Verdict is at or past its noisy ceiling (best judge κ = 0.620 against human κ = 0.562, i.e. the judges beat the humans, which is a statement about the labels rather than the judges). Arena seed perturbation moves accuracy only 0.004–0.032, smaller than the robust 1.7B→4B Arena gap. Prompt-sensitivity checks preserve cross-family ordering, with stronger judges more prompt-stable.

Weight it accordingly. Two model families, two reference judges, four datasets, a fixed main prompt with prompt sensitivity as a robustness check only, one primary decoding regime per experiment. The MiniMax axis is an observed release sequence, not a controlled ablation — a null result on it is a null result about released APIs, not about training interventions. McNemar tests are adjacent-pair tests and constitute no formal between-axis comparison.

The instrument under the judge — validating rubric items#

Everything above validates a judge. CalibratedRubric (Chen et al., FinStep + StepFun, arXiv 2607.29252, July 2026, empirical) validates the thing the judge is pointed at, and the distinction it draws belongs in this page's vocabulary because the field routinely conflates all three:

Property of a rubric item	Observable	What it does not establish
Measurability	inter-judge agreement rate	that the criterion is worth measuring
Informativeness	IRT item information over the fitted ability range	that graders can apply it consistently
Validity	expert endorsement	— the paper supplies no automated substitute

Their z_j is defined as reproducible judgeability and immediately qualified: "necessary but not sufficient for substantive expert endorsement." That is this page's reliability-is-not-validity thesis restated one level down — an item every judge agrees on can be an item that measures nothing anyone cares about, and agreement is exactly the statistic that will not tell you.

Where it collides with Finding C. CalibratedRubric's measurability posterior is a Beta–Bernoulli over the fraction of instances on which all judges agree, and it requires ≥ 3 judges to carry any signal at all (with two, unanimity is definitionally pairwise agreement — their two-judge HealthBench and HelloBench arms show no gain). Assumption A3 concedes the threat without measuring it: "LLM judges share training data and conventions, so correlated errors can inflate apparent consensus and estimated sensitivity or specificity." Yang's ρ = 0.944–0.972 / 0.664–0.706 is the magnitude of exactly that, measured under no optimization pressure. The two results are not in conflict — they concern different uses of redundancy (Yang: extra jurors as votes, capped by ρ; Chen: extra jurors as an observation about the item), but ρ threatens both. The seam shows in Chen's own data: posterior measurability predicts agreement at r = 0.589 / 0.558 on the LLM-judged blocks and drops to r = 0.127 on JudgmentBench, the single block whose gold labels are human. The mechanism is best-evidenced where LLMs are grading LLMs.

And a directional judge–human bias that agreement filtering does not remove. On JudgmentBench the LLM judges assign positive labels at 55.6–62.9% against the human gold's 47.1% — the authors' own words, "a systematic judge–human mismatch that measurability filtering does not fully eliminate." Filtering for agreement makes the panel more self-consistent without moving it toward the humans. Same direction as Reference-Free Judge Over-Crediting's over-crediting, arriving from rubric grading rather than reference-free QA. The MVVP-shaped addition: when a rubric bank is built from judge agreement, report the panel's base positive rate against the human base rate, because a filter that improves κ and leaves that gap intact has improved reliability without improving calibration.

Who judges well (provider patterns)#

• Frontier/reasoning models reduce position bias but don't eliminate it. Range spans ~two orders of magnitude: Gemini 2.5 Pro 0.002 (best) to Qwen 3 8B 0.192 (worst); within the Gemini family, 2.5 Pro (0.002) vs 2.5 Flash (0.125) differ 70×. The pre-registered prediction that all three thinking-architecture judges (GPT-5.4, Gemini 3.1 Pro, DeepSeek V3.2) would fall below 0.05 held for only Gemini 3.1 Pro (0.038); GPT-5.4 (0.083) and DeepSeek V3.2 (0.094) missed.
• Anthropic judges post the strongest joint performance on hard items — average JudgeBench κ = 0.770 (Opus 4.6 0.875, Sonnet 4.6 0.782, Haiku 4.5 0.653) at the lowest cohort-level position bias of any provider (0.020). OpenAI flagships (GPT-4o/4.1/5.4) average JudgeBench κ = 0.467; generational progress is legible on JudgeBench (0.309 → 0.487 → 0.606) but nearly invisible on the compressed MT-Bench scale (0.451/0.451/0.457).
• Mid-tier can beat frontier on a specific axis. Kimi K2.5 records the lowest position bias of any non-Gemini judge (0.004) and JudgeBench κ = 0.720 at a fraction of frontier cost — so "pick the strongest model as judge" is not a safe default; pick on the dimension you care about.

An eval-hygiene lesson: RewardBench was silently degenerate#

The authors predicted RewardBench would produce κ ≈ 0 because the standard generative loader places every chosen response in position A — making the human label identically "A", collapsing p_e, and degenerating Cohen's κ to 0.000 for every judge. Per-item position randomization (seed 42) restored a valid signal (κ ∈ [0.616, 0.898]), refuting their own hypothesis. The transferable warning: a fixed answer-position convention in a benchmark loader can silently zero out your chance-corrected metric — an artifact of the harness, not the judge.

The Minimum Viable Validation Protocol (MVVP)#

Before deploying an LLM judge:

1. Chance-correct. Report Cohen's κ (or Krippendorff's α) alongside any exact-match figure, and treat the chance-corrected metric as the headline reliability number.
2. Swap positions. Measure position bias via paired AB+BA evaluations; report |P(A wins) − 0.5|.
3. Replicate. Measure test-retest over ≥ 3 independent runs at temperature 0 with response caching disabled.
4. Cross-validate. Evaluate on ≥ 2 benchmarks spanning preference-style and correctness-style label distributions.
5. Audit the paradox. When test-retest exceeds 0.95, verify position bias is below 0.10 before claiming reliability. High stability with high bias is a failure mode, not a strength.

The paper flags its own partial-adoption risk: reporting κ alone (step 1) without the position-swap and consistency checks can manufacture a false sense of having addressed judge reliability.

Scope caveats (from the paper's own Limitations)#

The findings are a snapshot, not a universal law: English-only, text-only, three established benchmarks, a single pairwise rubric template, and a five-week window (hosted endpoints drift silently, un-re-measured here). Thinking channels were suppressed for all reasoning-capable judges to keep them comparable — reasoning-on could change every agreement/consistency/bias profile. And calibration proper (Expected Calibration Error, Brier score) is deferred because most providers don't expose token logprobs — so the question of a judge's confidence calibration remains open.

The same failure outside evaluation: taxonomy classifiers#

Google ATLAS (July 2026) reproduces this page's core structure in a different domain — LLM classifiers mapping conversations onto BLS/O*NET/ATUS categories for economic research rather than scoring model outputs. The shape is identical: a lenient validation metric flatters the classifier, and which metric you report decides whether the pipeline sounds trustworthy. Where this page finds exact-match agreement overstating chance-corrected κ by 33–41pp, ATLAS finds human approval (85.8%) overstating exact accuracy (22.6%) by 63pp at the O*NET task level — because asking a rater "is this label defensible?" is a much easier question than asking them to pick the right one from 18,797.

Two transfers run both ways. ATLAS randomizes classifier option order to defeat position bias — a direct application of the consistency–bias lesson here. And ATLAS supplies a ceiling this page's protocol lacks: human annotators disagree with each other on 42–48% of 3-digit occupation codes (Mellow & Sider 1983; Mathiowetz 1992), so accuracy measured against 100% is measuring the wrong target. A Minimum Viable Validation Protocol arguably needs a human-ceiling estimate alongside its chance correction.

Right metric, wrong axis — and a ratification rate reported as agreement#

A second taxonomy classifier supplies the failure the MVVP has no step for, and it is instructive precisely because the metrics are good. Pahuja, Brokman, Hofman et al. (An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures, arXiv 2608.03735, 2026-08-04, empirical) use Claude Opus 4.8 to sort agent planning failures into six categories, and validate it better than most: chance-corrected κ = 0.860 reported alongside macro-F1 = 0.906 rather than raw agreement (step 1, done properly), uncertain annotations excluded rather than coerced, and a per-category breakdown published including its own weak cells — Operation 83.3%, Answer format 85.7%, Other 88.2% against Entity 94.4% and Source/Temporal 100%, overall 88.9% on 117 decisive verifications.

Two defects survive all of that.

The validation is stratified by category; the claim varies by language. The paper's headline diagnostic result is a per-language gradient across eleven languages spanning five orders of magnitude of Common Crawl share, and no per-language agreement figure appears anywhere — even though the annotation sample was deliberately drawn to span the resource spectrum, so the cut is available and simply was not made. The two lowest-agreement categories are two of the three whose shares move most along the gradient. Its own Limitations concede that "judge calibration may differ across languages." Nothing in the MVVP's five steps catches this: they govern the metric, position bias, replication, benchmark spread and the reliability–bias paradox, and none of them says stratify your validation along the axis your headline claim varies on, and report the per-stratum numbers. The page already praises User Awareness's Appendix G for stratifying and reweighting to the population; this is the counterexample that turns that praise into a rule, and a candidate sixth step. Its coverage is also stated three mutually incompatible ways across §5.2, App. G.4 and Table 11's caption (6 annotators / 122 samples / ~14% versus seven annotators / 280 samples / ~30% versus 117 decisive with 5 annotators and six exclusions) — the kind of accounting slip that only matters because it is the number licensing the judge.

And 88.9% is a ratification rate, not an agreement rate. The annotation guide presents each sample with "all inputs supplied to the judge LLM together with the judge's response, and asks annotators to verify the assigned category." That is the second corpus instance of the asymmetry this page already flags on Automated Failure Attribution, where a human panel ratifies pipeline labels at 94.0 / 90.0 / 90.0 while models predict from scratch — and two instances make it a pattern worth naming rather than a quirk of one benchmark. The reason it matters here is that the mechanism is measured on the sibling page: judging a shown candidate is candidate anchoring, and Zhou's de-anchoring result puts the size of it at FPR 0.719 → 0.012 once the judge must commit its own answer before seeing the candidate. Nobody has run that control on human validators, and the fix is the same shape and nearly free: have a subset of annotators label blind, then reveal the judge's verdict, and report both numbers. Until that exists, a ratification rate is an upper bound on validator-judge agreement of unknown tightness, and every taxonomy-classifier study in this corpus reports the upper bound.

The ceiling as the target, and two construct-validity checks — from a production deployment#

Everything above is measurement of judges by people whose job is measuring judges. Shopify's Sidekick account (McNamara & Mazza-Anthony, Shopify Engineering, 2026-08-05, case-study) is a team validating a judge because it is about to become an RL reward, and its protocol contains the step this page has been asking for plus two nobody here has proposed. Evidence first: this is a first-party account of the authors' own system with no replication and no adversarial review, so it cannot upgrade or contradict any empirical finding above. What it contributes is practice, and two candidate protocol items — checkable on their own terms regardless of who ran them.

The human ceiling, measured in-house rather than borrowed. Before any judge exists, two expert annotators blind-annotate 25 randomly sampled production conversations and their agreement is recorded with Cohen's kappa — chance-corrected, exactly as step 1 prescribes, applied one level up to the humans. The threshold is a rewrite trigger rather than a pass mark: at κ ≈ 0.2 the rubric is declared ambiguous and iterated, on the argument that "if the rubric confuses several product experts who work on this product every day, it will confuse an LLM too." Then the judge is aimed at that number instead of at 100%: "that agreement is the judge's ceiling… The goal is not a judge that is 'perfect,' but one that matches humans about as well as humans match each other." Their reported result, which exists only inside a figure and nowhere in the article's text, is human agreement 83% — annotated in the chart as "perfect: unreachable" — against judge 80%.

This is the ATLAS-derived note above ("a Minimum Viable Validation Protocol arguably needs a human-ceiling estimate alongside its chance correction") arriving as a shipped step, and it improves on ATLAS in one respect worth naming: ATLAS's ceiling is borrowed from the literature (human annotators disagreeing on 42–48% of 3-digit occupation codes, Mellow & Sider 1983; Mathiowetz 1992), where this one is measured on the actual rubric, by the actual annotators, on the actual traffic, before the judge is built. A ceiling estimated off someone else's task tells you a ceiling exists; one estimated on yours tells you where to stop optimizing.

And the defect is this page's own Finding 1, in the wild. The κ discipline is applied to the annotator pair and then dropped: the 83%-versus-80% comparison — the number the whole ceiling argument rests on — is raw agreement, uncorrected, one paragraph after the team correctly reached for chance correction. Finding 1 says the gap between the two tracks the label distribution, and the figure shows a Pass / Borderline / (implied fail) scale, i.e. the multi-category regime where deflation was largest in this study's own cohort. The direction is certain and the magnitude is not computable from what is published (the label distribution is not given), so the honest statement is: the ceiling framing is right and the metric it is expressed in is the one this page demotes to a secondary figure. Note also what the account never says — the κ at which they shipped. Only the rewrite trigger (≈0.2) is published, so the rubric's final chance-corrected agreement is unknown.

The two additions, and why they belong in a validation protocol rather than in a product doc. Every MVVP step and every Yang audit item measures a property of the judge in isolation. These two measure whether it is pointed at the right thing:

Check	Procedure	What it establishes that reliability cannot
Online-metric backtest	Replay the judge against previous A/B tests and ask whether it recovers the direction of known wins and losses in engagement, retention, or whatever the product is built to drive	That the offline proxy is correlated with the outcome it is a proxy for. A judge can be κ-clean, position-unbiased, replicable, and still rank the losing variant first
Targeted degradation test	Deliberately make one behaviour worse — offline or on a controlled traffic slice — and confirm the corresponding criterion falls, and specifically that one ("if the system stops trying to fulfil the user's goal… the goal-fulfilment score should fall specifically")	That the rubric's criteria are separable and each one measures what its name says. This is the per-criterion analogue of a manipulation check

Both are construct-validity instruments, which is precisely the half this page's title says the field leaves out — and the backtest is the only device in the corpus that grounds a judge in an outcome the judge did not define. Their stated precondition is a fourth prescription that reads as an aside and is not one: "keep each judge small and targeted rather than cramming all of your product's behavior into one… focused judges make these tests easier to interpret and the resulting metrics easier to trust." A degradation test on a blended judge is uninterpretable by construction, so judge decomposition is what makes criterion-level falsification possible at all — the same move Agent Quality Flywheel arrives at from the metric side (promote one concern to one stable metric) rather than from the validation side.

What none of this touches is the failure Reference-Free Judge Over-Crediting measures, and this deployment is the corpus's most exposed instance of it: the same calibrated judge becomes the hard-negative selector, the repair gate and the GRPO reward, and nothing in the account re-measures its agreement once a policy is being optimized against it. A one-shot validation, however good — and this is a good one — certifies a judge that will be true only until something starts pushing on it.

Connections#

• Skill Lift — a live instance of the gap this page names, in a benchmark being used as a distribution gate. NVIDIA publishes five 0–100 rubric dimensions across 300+ skills and never names the grader, reports no human-agreement or chance-corrected statistic, and runs no position-bias check; a +41-point delta measured that way inherits whatever reliability an unvalidated judge has. The Minimum Viable Validation Protocol is exactly the missing disclosure

• Agent Quality Flywheel — the deployment this section is drawn from, and the reason its judge matters more than most: after calibration the judge becomes the offline metric, the hard-negative selector, the replay-repair gate and the RL reward all at once, on a daily training cadence. It also supplies the corpus's clearest statement of why the rubric comes before the judge — "defining quality… begins as a specification of what good looks like and becomes the reward signal that drives learning"

• User Awareness — the same organization doing it right, two days later, which is what makes the contrast worth keeping. Its Appendix G validates the awareness judge on 74 blind hand-labelled items drawn from 11,908 runs, stratified across judge-positives, lexicon-flagged negatives and double negatives, balanced across model families and reweighted to the full population, reporting precision 0.73 [Wilson 0.56–0.85] / recall 1.00 [Jeffreys 0.42–0.99] for evaluation awareness and 0.94 [0.81–0.98] / 0.98 [0.74–0.99] for user awareness, plus an error analysis locating all 9 false positives in one task's vocabulary. Two lessons. The recall intervals are wide enough to matter — correcting the headline rates at the pessimistic corner still leaves them near 1–3%, so the validation is what licenses the paper's central negative claim rather than decorating it. And the same study supplies a validity threat no protocol here tests: a judge's score moves with who it believes is asking

• Misalignment in Production Agent Traffic — this page's failure mode in a result the corpus otherwise leans on. Transluce's entire 8,600-session misalignment rate rests on one unvalidated LLM judge: rubrics drafted by Claude Fable, refined by spot-checking 100 random runs, compared against "a small hand-labeled set" of unstated size, and screened by a GPT-5.6 Sol skeptical verifier — a sensible pipeline that yields no agreement statistic, no kappa, and no reported false-positive or false-negative rate. The stated design bias toward avoiding false-positive severe cases is the only quantitative-ish handle on it. A textbook instance of the under-rigour measured here, on a load-bearing number

• Usage-Telemetry Classifier Validation — the same structural failure in AI-usage economics: a lenient metric (human approval, 85.8%) flattering an LLM classifier whose exact accuracy is 22.6%, plus a human-disagreement ceiling this page's protocol doesn't estimate

• Matched Comparisons for Memorization Claims — the same arithmetic, one field over. Chance-correction is "observed agreement minus the agreement you'd get by chance"; Cooper et al.'s calibrated memorization rate is M(τ) = G(τ) − FPR(τ) — observed generation on training data minus the generation rate matched non-training data reaches. Both correct a raw rate by a baseline that the field had been assuming was zero, and in both the correction is large (κ deflation of 33–41pp there; ~24% of the apparent extraction rate at 10-token suffixes there). The memorization version has the cleaner null — non-membership is definitionally exclusive, so its baseline is measured against something that cannot be a true positive, where a judge's chance baseline must be modeled

• LLM-as-a-Judge — the primitive this page validates; kappa deflation and the consistency–bias paradox are the reliability failures its DRACO-style protocol can hide. It also hosts the upstream instrument: CalibratedRubric's rubric bank, where measurability, informativeness and validity separate and the field's unanimity filter turns out to decay exponentially in leaderboard size for reasons unrelated to rubric quality

• DRACO Benchmark — the judge-model-invariance counterpart; DRACO's "use rankings" reassurance is bounded by this paper's benchmark-variance (two different invariances)

• Production-Sourced Evaluation — the orthogonal axis of eval quality: production-sourcing fixes task representativeness, this page fixes grading validity; a representative task graded by an unvalidated judge is still untrustworthy

• Automated Failure Attribution — a benchmark that follows most of this protocol, applied to label quality rather than judge quality: three independent annotators on 100 stratified traces, majority vote, a chance-corrected Fleiss κ = 0.73 rather than raw agreement, an explicit "no clear decisive error" escape hatch (2.0%), and a published transfer matrix showing which families annotators moved labels between. The gap worth naming is one this page's protocol would catch: the human panel ratifies a supplied label while the models predict from scratch, so its 94.0 / 90.0 / 90.0 is not a like-for-like human ceiling against the models' 73.9 / 57.5 / 22.2 and should not be read as one. Extended 2026-08-13: that page's second source repeats the ratify-don't-predict design (annotators shown the judge's own inputs and its verdict, asked to verify the assigned category) and adds the failure this protocol has no step for — validation stratified by category while the headline claim varies by language, developed in the taxonomy-classifier section above

• Optimizer–Evaluator Decoupling — decoupling makes the evaluator independent but not valid; a decoupled-yet-reproducible judge can be maximally biased, so the MVVP is a concrete answer to that page's "what verifies the verifier?" regress

• Evals as Product Spec — "ten great evals" graded by an LLM judge inherit this validation debt; authoring a good eval and validating the judge that grades it are separate disciplines

• Automated Behavioral Audit — the highest-stakes judge deployment in the vault: a judge model scoring safety behavior across dozens of dimensions, feeding thresholded RSP determinations — exactly where inflated agreement and position bias would corrupt a ship/no-ship call

• Verification as the New Bottleneck — LLM-judge validation is the quality-control layer under one imperfect answer to verification-at-scale

• LLM-Assisted Grey-Literature Theory Building — a research pipeline that practices this page's core prescription: it reports the chance-corrected Cohen's κ = 0.75 (not raw exact-match agreement) against a stronger re-judging model as the reliability number for its relevance judge — the MVVP's step 1 applied to a corpus gate

• Motivated Mislabeling — the failure this protocol structurally cannot surface: a judge that labels for the consequence of the label is content-dependent and direction-consistent, so κ, test-retest and position-randomization all read clean; the only control that catches it is varying the stated consequence

• AI-to-AI Coercion — a design that removes the judge from the headline scoring path entirely: the escalation rung is a required tool argument the model under test emits about its own message, so the primary measure has no validation debt at all (a label-neutralization control rules out a demand effect); judges are kept only for the secondary free-text ladder and the two-judge fabrication adjudication

• Stopping Under a Noisy Verifier — the same statistic doing a different job, and the case where this page's prescription backfires. Youden's J is a judge-selection metric here; there it is a deployment parameter that decides whether an agent loop can be steered on the judge's verdicts at all — a low-J judge's pass rate is ρ₀ + J·Q, mostly its own false-accept rate, so it can rise while true quality falls. Two transfers. It converts the MVVP's binary verdict ("this judge is not valid") into a graded operating decision: above J ≈ 0.18 a calibrated loop tracks the true-parameter reference within 2.8pp, at J = 0.03 it collapses by 58.0pp, and the fix is not a better judge but a rule that does not need the judge calibrated. And it is the corpus's counterexample to "measure more carefully" — the label-free mixture estimator that recovers ρ₀ and ρ₁ is identifiable iff J ≠ 0, so at low J a larger calibration sample makes the estimate strictly worse (ρ̂₁ 0.27 at N = 120 → 0.077 at N = 300, true 0.609). Their held-out separation test exists precisely to avoid diagnosing a broken estimator with its own output, which is step 5 of the MVVP arriving as a deployment gate

• Reference-Free Judge Over-Crediting — the multilingual, reference-axis sibling: a second reliability-without-validity instance (temperature-0, perfectly reproducible judges are systematically invalid without a reference), supplying a concrete answer to this page's "biases beyond position bias" open question and extending the evidence into low-resource languages this English-only study lacked. Its second source adds the axis the MVVP structurally cannot cover — validity is not a property of a judge, it is a property of a judge under a given amount of pressure. The same judges hold usable discrimination (0.21–0.38) when they are only rating, and collapse to 0.05–0.17 once a policy is optimized against them, with true accuracy unmoved. Every step of the protocol is a one-shot measurement of a judge nobody is pushing on, so a judge can pass all five and still be worthless as a reward; the check that catches it is not on the list — a held-out signal the judge never sees, re-measured over the course of the optimization

• Tool-Output Pruning — the case that runs this page's suspicion backwards, and locates the blind spot it shares with the field. Selecting a context pruner by per-line F1 — a deterministic, reference-backed, judge-free label-match metric — picks the wrong head: on both hand-examined cases F1 ranks a signature-only pruner above a usable one (0.53 vs 0.49, and 0.80 vs 0.71) while a GPT-5.4-mini judge separates them by 5-6 points (2/10 vs 8/10; 3/10 vs 8/10), and two loss functions match the winner's F1 while their judge scores collapse to 5.30 and 3.03. The mechanism is that F1 scores set membership on the kept lines while the task is whether the retained skeleton supports the agent's next action. This does not weaken the MVVP — it says the validation debt is not specific to judges: an unvalidated label-match proxy carries the same construct-validity gap, and nobody demands κ, position checks, or a human ceiling from a metric that looks arithmetic. Left open on that page: the judge doing the correcting here is itself unvalidated in this role

• Deep Research Agents — the high end of the agreement range, and what buys it. MisKnow-Agent's FCAR judge reaches 99.7% raw agreement and Cohen's κ = 0.993 against two blinded annotators whose disagreements were adjudicated to a single reference label, on a 300-report sample. The instructive part is the task shape: not "score this report" but "does this report's own conclusion, recommendation or executive summary endorse this specific claim?", with an explicit decision rule listing four ways of surfacing the claim (mention, quotation, attribution, hedging) that score as non-adoption. Reducing an open-ended report grading to one sharply-bounded binary is what moves κ from the 0.4–0.75 band typical of rubric grading to near-unity — a design lever for judge validity that sits upstream of every measurement in the MVVP

• The Verifiability Thesis — the horizon this page's discipline bounds. Karpathy's "council of LLM judges" is the construction that extends the reward signal into soft domains; the MVVP validates one judge, and Yang's ρ = 0.66–0.97 says the council's headroom over that one judge is small by measurement, before anything optimizes against it

• Benchmark Score Redundancy — where this page's debt gets inherited by a proof. DeepMind's CollabEval derives unbiased estimates with asymptotically valid confidence intervals for a model's benchmark mean at a fraction of the annotation budget — and on three of its five datasets the quantity being bounded is an autorater's output (GPT-4 Turbo win-rate on AlpacaEval, AutoAIS on AQA, MetricX on WMT24++). The interval is over sampling uncertainty in the mean of whatever the rater produced, so kappa deflation, position bias and reference-presence sensitivity all pass through it untouched: a 30%-tighter CI around a judge with κ = 0.48 is a more precise estimate of a badly-measured thing. The composition that would close the gap already exists in the same literature (stratified PPI relating cheap autorater labels to expensive human ones), but that paper does not build it — which makes "statistically efficient" and "validated" fully orthogonal properties of an eval pipeline

• How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them? — the cluster synthesis: unvalidated grading is one of the four corruption channels of the public benchmark number, and the MVVP is the validate-the-judge move in the five-part replacement portfolio

• Deterministic Engineering for Agent Code Review — a same-matcher defense caught in the act. Every figure in OpenCodeReview's twelve-configuration comparison is a Qwen3-235B-A22B-Instruct semantic-match verdict, and §6's mitigation — same matcher, same protocol, so relative comparisons hold — is this page's central distinction (reliability is not validity) restated as though it settled the question. No κ against human matching decisions, no position or order controls, and the five-run-mean variance control is a homogeneous jury: this page's ρ ≈ 0.66–0.97 says repeated sampling of one judge buys far less than independence would. What the paper does have is unusually strong on the other half — AACR-Bench's 1,505 ground-truth comments are expert-verified through three rounds by 80+ engineers — so the failure is squarely the MVVP's missing step 1, not the label quality underneath it

• Weak-Verifier Ensembling — the system this page's Finding C bounds, and the most direct collision in the corpus. Weaver's aggregation is derived from an explicit assumption that "each verifier captures an independent aspect of the correctness"; ρ = 0.944–0.972 (Qwen3) and 0.664–0.706 (MiniMax), both measured on homogeneous juries, say that assumption is false in the direction that costs it, and heterogeneous juries underperforming Poisson-binomial predictions removes the obvious hedge. Three things scope the collision rather than settling it — Weaver's pool mixes trained reward models with judges (never measured here), it filters and weights against real labels rather than voting reference-free, and it runs as a static selector rather than a reward. Its own reported non-monotonicity in ensemble size is the correlated-error signature showing through

• What the Instrument Can Resolve: Two Headline Numbers and Their Missing Denominators — the human-ceiling step, promoted from an aside to a reporting standard, and applied outside evaluation. It takes Yang's leave-one-annotator-out calibration and the Shopify pre-judge ceiling as the two answers to Usage-Telemetry Classifier Validation's open question, shows ATLAS already holds the annotations to compute one and stops at an incomparable pair (model-vs-plurality-of-three κ = 0.83 against human-only pairwise κ = 0.66), and reports the resulting (observed − chance) / (ceiling − chance) figures — 73–82% and 85–94% of the human ceiling where the raw numbers read 42.5% and 71.6%. It also supplies a fresh instance of the ratify-versus-predict asymmetry this page names (an approval rate is anchored, a disagreement rate is blind, so the two cannot be divided), and a load-bearing economics result running on a wholly unvalidated ten-way LLM classifier: the 25%/7% screen-out split in Controlled Variance: AI's Edge as Reduced Dispersion, whose codebook thresholds on a variable the treatment moves

• The Price of Mixing Agents, and the Principal Nobody Counted — this page's Finding C doing work outside evals: heterogeneous juries underperforming Poisson-binomial independence is the corpus's only direct test of provider-mixing as a decorrelation intervention, and it reports the direction with no coefficient, which is why no variance-vs-exploitation frontier can be drawn for agent populations. It also flags that Weak-Verifier Ensembling mis-renders the ρ pair as a within-vs-across-family contrast when both ranges are homogeneous-jury values

• AI-Assisted Error Analysis — the step upstream of anything this page validates. A judge cannot be validated against criteria nobody wrote, and Shankar's argument is that writing them is where automation stops paying: agents may apply a human-authored criterion across a corpus but should not propose the criteria, because validating an agent's proposed taxonomy costs more than authoring your own. The validation burden this page quantifies is the reason

Open Questions#

• The MVVP validates reliability and bias; calibration proper (ECE/Brier) is deferred for lack of provider logprobs. How far can a judge's absolute score be trusted for a threshold once confidence calibration is measurable?
• All judges were run with thinking suppressed. Does reasoning-on flip the consistency–bias paradox, or just move the numbers?
• Hosted endpoints drift silently between provider updates. How stable are these agreement/bias profiles over a longer horizon than five weeks — and should judge validation be continuous rather than one-shot? Partially answered (2026-08-04) by Yang et al. (2026), on the announced-upgrade sibling of the question rather than silent drift. Across four released MiniMax generations (M2 → M2.1 → M2.5 → M2.7) on four datasets, adjacent accuracy moves at most 0.022 and not one of the nine adjacent McNemar tests reaches even uncorrected p < 0.05 — and because those tests are paired on parse-shared examples, this is stability of the individual verdicts, not merely of the aggregate. So a deliberate version step at the top of the capability range is a small perturbation to the agreement profile. Three things keep this open. (i) These are version-labeled releases you can pin, not the unannounced same-endpoint drift the question is about — nobody has re-measured a fixed endpoint over months. (ii) Stability of accuracy is not stability of bias: position-flip rates still span 0.117–0.147 across the MiniMax releases, and no one tracked whether that band moves within a single version. (iii) The finding runs the other way on the parameter axis, where a step does move things and can move them down (Qwen3 1.7B → 32B costs 0.065 on Judge's Verdict), so "upgrades are safe" is not the lesson — "upgrades are a measurement event that must be re-validated, and the null case is the lucky case" is. That is an argument for continuous validation, from the direction of the one axis where it was cheap to check.
• Does the paradox generalize beyond position bias — i.e., are there other biases (self-preference, lineage) that high test-retest also masks? Partially answered by Kranti & Vajjala (2026): yes — reference-presence sensitivity is a large one. Their temperature-0 judges (thus perfectly reproducible) systematically over-credit incorrect answers in no-reference settings, an invalidity invisible to any reliability metric until a gold answer is added, which flips up to 85% of verdicts and lifts human-alignment sharply (e.g. Gemma3-27B 0.34→0.85 NR→RV). It does not close the question — self-preference and lineage remain unisolated (their design deliberately overlaps judge and generator but doesn't attribute the effect), and it is a different bias on a different (multilingual QA, three specific judges) setup, not a re-run of this study's position-bias protocol. The self-preference half then largely resolves negatively (2026-08-04), via Zhou (2026): errors optimized against a self-judge transfer to judges from other families that were never in the loop (Llama 0.480 → 0.568, Gemma 0.764 → 0.918) and to same-family judges 3.5× larger (still 77%), with a three-family unanimous-accept ensemble passing 55% and judge acceptances pairwise correlated at φ = 0.29–0.38. So lineage amplifies — the self-judge is the worst single cell at 0.906 — but is not the mechanism; the bias is a shared property of the candidate-conditioned channel. This also adds a bias class no reliability metric on this page can reach, because it is not a property of the judge at all: the same judge, unchanged, is valid before optimization (discrimination 0.31) and invalid after (0.09), so any one-shot validation — including the full MVVP — certifies a judge that will be true only until something starts optimizing against it. That is a direct argument for the continuous-validation question two bullets up.

Sources#

• Sidekick's continual learning loop — Andrew McNamara & Cody Mazza-Anthony, Sidekick's continual learning loop, Shopify Engineering, 2026-08-05, case-study (first-party account of the authors' own production system; no replication, no adversarial review, every figure self-reported — it cannot upgrade or contradict any empirical finding on this page). Cited for the pre-judge rubric protocol (two expert annotators, blind, 25 random production samples, Cohen's κ with a ≈0.2 rewrite trigger), the judge's-ceiling framing quoted above, the A/B backtest and per-criterion degradation test, and the keep-judges-small prescription. The 83% / 80% pair exists only inside the article's Judges' agreement figure — grep-confirmed absent from the page text — and was read directly under the image two-pass rule; both are raw agreement, not chance-corrected. Full source treatment and evidence handling on Agent Quality Flywheel and in wiki/sources.md

• An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures — Pahuja, Brokman, Hofman, Nizri, Vishna, Goldfarb-Tarrant, Marchisio, Kojima & Vainshtein (Fujitsu Research of Europe / Cohere / Fujitsu Research), An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures, arXiv 2608.03735, 2026-08-04, empirical. Cited here only for App. G (the six-category judge design, its inputs, and Claude Opus 4.8 as the judge model), App. G.4 + Table 11 (the annotation guide's verify-the-assigned-category instruction, the per-category agreement figures, κ = 0.860 and macro-F1 = 0.906), App. H + Table 12 (the eleven-language Common Crawl spread the unstratified claim runs over) and Limitations (the authors' concession on cross-language judge calibration). Table 11's per-category n sums to its own Overall row and 104/117 reproduces the printed 88.9%; the three mutually inconsistent statements of annotation coverage are quoted from §5.2, App. G.4 and Table 11's caption respectively. Full parse notes and COI on Automated Failure Attribution

• User awareness in frontier models — Zhong, Raghunathan, Laidlaw & Steinhardt, Transluce, 2026-08-06 (empirical): Appendix G — the validation this page prescribes, run on an awareness judge: 74 blind hand-labelled items from 11,908 readable-reasoning runs, stratified across judge-positives, lexicon-flagged negatives and double negatives, balanced across families and reweighted to the population; VEA precision 0.73 [Wilson 0.56–0.85] / recall 1.00 [Jeffreys 0.42–0.99], VUA 0.94 [0.81–0.98] / 0.98 [0.74–0.99]; raw per-setup agreement 32/32, 15/18, 18/24 (VEA) and 31/32, 17/18, 22/24 (VUA); and the error analysis placing all 9 VEA false positives in grading, where the task's own vocabulary was misread as evaluation awareness. Also the identity-conditioned grading result, a validity threat no protocol here tests. Full treatment on User Awareness

• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — Norman, Rivera & Hughes (UC Berkeley, arXiv 2606.19544, June 2026), empirical. §2.1 (metric definitions: kappa deflation Δκ, consistency–bias paradox), §4.1 (kappa deflation universal, Table 2), §4.2–4.3 (position-bias heterogeneity, cross-benchmark rank instability), §4.7 (the paradox — Qwen 3 8B, Gemini 2.5 Flash), §4.8 (verbosity bias < 0.011), §4.9 (provider families), §4.5 + App. E (RewardBench position-randomization fix), §5.3 (MVVP), Limitations + App. H (partial-adoption / temporal-drift risks)

• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability — Zongyou Yang, Yinghan Hou & Xiaokun Yang (Imperial College London + Nanchang Institute of Technology, arXiv 2607.08535, 2026-07-09), empirical. §1 (evaluator-replacement ambiguity, RQ1–RQ3), §3.2–3.3 (eight-judge two-axis panel, four datasets, T = 0.1, parse-shared McNemar, Holm over 18 tests), §4.1 + Table 3 (upgrade non-interchangeability; the 1.7B→4B step as the only Holm-surviving gain; PandaLM and Judge's Verdict regressions; no judge wins all datasets), §4.2 (position flip 0.320 → 0.117–0.147, verbosity 0.547 → ≈0.13 under a padding probe, Pearson r = −0.957, M2.7's residual 14.7%), §4.3 (ρ-corrected beta-binomial; ρ = 0.944–0.972 Qwen3 / 0.664–0.706 MiniMax; K = 1,3,5 → 0.463/0.475/0.482; heterogeneous juries also sub-independence), §4.4 (debate shifts +0.243 to +0.317 with no parser audit trail), §4.5 (human ceilings, Arena seed spread 0.004–0.032), Table 4 (the minimal audit trail), §6 (threats to validity)

• Parse note. Ingest flagged one table-collapse cell; the corpus-wide pattern held — Tables 2, 3 and 4 each had every data row welded into a single grid row (3, 4 and 6 rows respectively), which the checker surfaced for Table 3 only. All three were recovered with pdftotext -f 4/5 -layout against the local PDF and reconciled cell-for-cell: value order inside each collapsed cell is correct, no digit is wrong, nothing was dropped — only the row mapping was lost. Table 3 as quoted above is the recovered version. Table 1 parsed clean.

• Figures read from the page images per the two-pass rule. Figure 2 supplies the per-model LLMBar accuracies (Qwen3 0.463/0.617/0.647/0.678; MiniMax 0.829/0.839/0.832/0.832; GLM-5.1 0.900, mimo-v2-pro 0.883) that reframe the MiniMax null as a ceiling effect — this is not in the prose, which reports only adjacent differences. Figure 5c confirms the per-judge ρ values quoted from prose. Figure 3's McNemar grid gives the uncorrected p-values (Qwen3 1.7B→4B: LLMBar and Arena p <.001, PandaLM 0.090; 14B→32B PandaLM 0.032, which does not survive Holm).

• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation — Mengting Chen et al. (FinStep + StepFun, arXiv 2607.29252, 2026-07-31), empirical. Cited here for the item-level half only (full treatment on LLM-as-a-Judge): §2.1 (z_j as reproducible judgeability, explicitly short of expert endorsement), §3.3 (Beta–Bernoulli measurability posterior; the two-judge degeneracy), §4.2 (κ 0.604 → 0.743; r = 0.589/0.558 vs 0.127; positive-label rates 55.6–62.9% against human gold 47.1%), §2.4 + App. B.5 A3 (correlated LLM judges as an acknowledged, unmeasured threat to the agreement posterior)