LLM-Judge Self-Preference

Sources#

• Self-Preference of AI Judges

Summary#

Self-preference is the judge-side form of the lineage problem: an LLM judge rating its own (or its family's) output more favourably than an independent rater would. The wiki carried it as a caveat with almost no numbers behind it. LLM-as-a-Judge listed it as a validity threat. GDPval reported that OpenAI's GPT-5-high grader agrees less with human experts on capable OpenAI models' outputs. Mirhoseini said models "like their own generations" and knew of no study (Weak-Verifier Ensembling). A practitioner survey stated "it agrees with itself" as a flat rule. Same-Model Review Blindness measured the detection-task cousin, where lineage shows up as seeing less.

Arena (Self-Preference of AI Judges, blog post, 2026-09-30, empirical) supplies the first per-model measurement on current frontier models, on real user prompts, against a human baseline on the same items. 12 judges × 1,460 English Text Arena battles (1 Jun–23 Sep 2026). Model names and the human vote were hidden. Each judge saw every battle twice with the answer order swapped, and the two readings were combined by a stated rule: the same answer both times is a pick; a pick plus a tie or "both bad" counts the pick; different answers count as half a pick (no pick in the shared-taste analysis); no pick either time drops the battle from the self-preference and family charts. Of 35,040 planned verdicts, 34,580 were valid. The rest were unusable replies, content-filter blocks or failed calls.

Evidence note. empirical, tier kept. The measurement is on Arena's own platform data, with a stated protocol, a position-swap control, 95% intervals on the per-model charts, and a human baseline on identical items. It is not vendor-claim, because the subjects are twelve third-party models, not Arena's product. The COI is real and points one way: Arena sells human-preference evaluation, and the post's closing advice, "check verdicts against real human preferences", recommends what Arena sells. Gaps: no released data, no judge prompt, per-model battle counts not given, the human side is one crowd voter per battle with no inter-human agreement ceiling, and how human ties are scored in the agreement analysis is never stated (see below). The same organisation's HarnessTax study (HarnessTax: How Much Does the Harness Matter for Coding Agents?) is filed empirical on the same grounds.

The headline, and the part of it that is decisiveness#

The headline number mixes two effects. "Models backed themselves about 70% more often than humans": on average a judge picked its own answer 58% of the time, people 34% (58/34 = 1.71, a relative ratio, not 70pp). People called a tie or "both bad" in 32% of battles, and every judge was more decisive than people, by +3.0pp (DeepSeek V4 Flash) to +27.9pp (GPT-5.6 Sol, 96% decisive against 68%). Some of the 24pp all-battle gap is therefore judges picking anything more often, not picking themselves.

The post's per-model chart (Fig 2) "sets ties aside", and that is the cleaner self-preference measure. Recomputing it from the Fig 1 bars as own picks over all decisive picks reproduces Fig 2 to within ~3pp (Astra ~53 vs 55.3, Sol ~38 vs 39.7, Luna ~32 vs 31.8, Fable ~35 vs 37.7, MiniMax ~−20 vs −21.8; wiki arithmetic on bar readings marked ~ in the transcription, before the half-pick rule). So Fig 2 is a decisive-verdict share. The Fig 2 mean is +16.0pp (wiki arithmetic), about two-thirds of the 24pp headline gap. The two numbers sit on different denominators, so they cannot be subtracted cleanly, but the 70% headline overstates the self-preference component.

On the raw Fig 1 shares, Astra called its own answer better in 88% of its battles and the opponent's in about 2%. The people who wrote those prompts picked Astra 29%, the opponent 36%, and called a tie or both bad in the rest. Fable picked itself 72% against 38% for people. MiniMax picked itself 22% and the opponent 62%, where people split 32/37.

So self-preference is a per-model property, not a law of LLM judges. Half the roster shows a large effect and half shows none detectable, and one model leans the other way. The flat rule "don't let the same model grade itself, it agrees with itself" (recorded as folklore on LLM-as-a-Judge) is right as a precaution and wrong as a description: it holds for the six top models here and fails for the other six. The four strongest self-preferrers are the three OpenAI models and Fable. Whether self-preference tracks capability or lab cannot be separated on twelve models from seven labs.

A figure discrepancy, logged and not resolved. The judge × author heatmap (Fig 3) has its own diagonal, and it differs from Fig 2 by up to ~4pp: Opus +23 vs +24.1, Kimi +25 vs +23.1, Fable +39 vs +37.7, DeepSeek V4 Pro −8 vs −3.9, DeepSeek V4 Flash −6 vs −7.1. Astra (+55), Sol (+40) and Luna (+32) match to rounding. The page does not explain the gap; plausibly the two charts score battles differently (for example, Fig 3 counting a judge's picks across all battles an author appears in, under a different tie or half-pick treatment). This page cites the Fig 2 and text values and uses Fig 3 only for off-diagonal family and taste patterns, where the differences do not change any sign.

Family loyalty: strong for OpenAI, mild for Anthropic, absent for DeepSeek#

The heatmap asks how much more or less often than people each judge picked each author's answer.

• OpenAI. Astra, Sol and Luna rated each other's answers +37pp above people on average (off-diagonal cells +20 to +50). The other nine judges rated OpenAI models only +6 above people. The OpenAI trio rated everyone else 15 points below people, down to −42 (Astra on DeepSeek V4 Flash) and −30 to −34 on Gemini.
• Anthropic. Fable and Opus rated each other +28 (Fable→Opus +21, Opus→Fable +34) against +15 from other judges.
• DeepSeek. The two DeepSeek models rated each other 16 points below people, close to the other judges' −20. No loyalty.
• No national loyalty. Chinese judges were 6 points more generous than people to American answers and 6 points harsher on other Chinese labs. American judges matched people on other American labs and were 12 points harsher on Chinese ones.

This is the scoring-task counterpart of Same-Model Review Blindness's detection-task result, and the two agree on direction. Both measure a judge acting in its family's favour: on a preference task the judge rates the family's output higher; on a bug-finding task it finds fewer of the family's defects. Neither is consistent across labs. Greptile had two families; here the effect is large for one lab, half the size for a second, and absent for a third.

Against Cross-Model Error Entanglement's "vendor is a weak index". Kohli measures a same-family excess in error correlation of only +0.047 φ on labelling tasks. That is not a contradiction. It is a different quantity: correlated errors on third-party items versus a directional preference for the family's own outputs. Kohli's own pairwise-preference task (RewardBench) had the largest same-family excess (+0.109), and Arena's task is pairwise preference. Resolution: vendor is a weak proxy for whether two judges' errors are independent, and a strong one, for OpenAI judges, for whether a judge will favour an author.

Shared taste: judges agree with each other more than with the voter#

Beyond family, the judges share preferences people do not. All ten judges outside Anthropic rated Fable above people, by +20 on average (wiki check: the Fable column excluding Fable and Opus averages +20.2). But the three OpenAI judges sit at +1, +3 and +8 while the other seven sit at +23 to +32, so the agreement is wide and uneven. Every outside judge was more generous than people to Kimi. Every judge, MiniMax included, rated MiniMax answers below people (−17 to −34).

Fig 5 compares, for each judge, how often it picked the same winner as the other AI judges with how often it picked the human voter's winner, excluding battles the judge wrote. The average judge sided with the other models 79.4% of the time and with the person 56.9%. Every judge sided with the AIs more, by +18.0 (Astra) to +26.7 (GLM 5.3: 85.2% vs 58.5%). In the 393 battles where every AI judge picked the same answer, the person agreed 60.6% of the time. Astra, Sol and Luna were the only judges to agree with people less than half the time (48.2%, 48.7%, 49.6%), and they also agreed least with the other AIs: an OpenAI taste of their own, not crowd-following.

Two construct problems the post does not address (wiki reading).

1. The human side may be capped by ties. People picked a winner in only 68% of battles. If a human tie or "both bad" counts as disagreement with a judge that picked a winner, which is what "did it pick the same answer as the person" suggests, then human agreement cannot exceed ~68% for a judge that always picks. 56.9% would then be most of the reachable range, and 60.6% on unanimous battles would be close to the ceiling. The post never says how human ties enter. Until it does, "AI judges agree with each other, not with us" is partly the decisiveness gap above measured a second time.
2. A consensus versus a single voter. "The other AI judges" pools eleven models' verdicts; "the person" is one crowd vote. Agreement with a pooled signal is mechanically higher than agreement with one noisy rater, even if both track the same quality. The comparison that would isolate taste is judge-vs-judge pairwise agreement against human-vs-human pairwise agreement on the same battles, the inter-rater ceiling LLM-Judge Validation argues every judge report should carry. Arena has repeat-voted battles and does not report it.

Neither problem touches the self-preference and family results, which compare each judge with humans on the same item and denominator, and set ties aside.

Read with those caveats, the 393-battle result still lands where Cross-Model Error Entanglement's unanimity finding does: unanimous AI agreement is a measurement of shared prior, not of correctness. On Kohli's MNLI items, nine-judge unanimity was 90.9% accurate against 100-annotator gold. Here it matched a single voter 60.6% of the time. The two numbers are not comparable, because the gold standards differ, but both say the same about using unanimity as a confidence gate.

Arena's prescriptions#

The post closes with four rules for anyone using models to judge models, all consistent with the evidence above:

1. Don't treat the author as a neutral reviewer. It will likely favour itself (true for 6 of 12 here).
2. Don't assume judges from other labs speak for your users: they agree with each other and still differ from people.
3. Expect judges to force a winner where people would call it close, and track each judge's decline-to-choose rate.
4. Check verdicts against real human preferences or expert views (the COI-aligned one).

A wiki addition: choose the judge per author, not per task. On Fable answers, an OpenAI judge and an Anthropic judge differ by 31–38pp (Fable column: Astra +1, Sol +3, Luna +8 vs Opus +34). On Gemini answers, OpenAI judges sit at −30 to −34 while the rest sit at −5 to −12. A single fixed judge over a multi-vendor leaderboard therefore imports a different bias for every row. This is the Version-Dependent Judge Error problem across authors rather than across releases.

Connections#

• LLM-as-a-Judge — the primitive this page measures a bias of. Its "self-grading and lineage bias" limit and its open question on when lineage bias flips a result get their first per-model, current-generation numbers here, on a scoring task rather than Greptile's detection task
• Same-Model Review Blindness — the detection-task form of the same coupling. There lineage shows as lower recall on the family's bugs; here as higher preference for the family's answers. Same direction (in the author's favour), and in both cases the size depends on the lab
• LLM-Judge Validation — the validation discipline this study half-follows. It controls position bias by order swap and compares against humans on identical items, but reports agreement against a single crowd voter with no inter-human ceiling, the reporting error that page's MVVP exists to prevent
• GDPval Benchmark — the earlier field sighting: OpenAI's own GPT-5-high grader agreed less with experts on capable OpenAI models' outputs. Arena's +37pp OpenAI family loyalty, on a newer OpenAI generation and an outside platform, is the same effect at a larger measured size
• Cross-Model Error Entanglement — agreement among judges as shared prior: 79.4% AI–AI against 56.9% AI–human, and 60.6% human agreement on unanimous battles. Also the reconciliation above: vendor weakly indexes error correlation but strongly indexes OpenAI judges' directional preference
• Weak-Verifier Ensembling — Mirhoseini's uncited "models like their own generations", now measured: true for six of twelve frontier models, false or reversed for the rest. A cross-lab verifier pool is the right instinct; this says which labs' verifiers carry the bias
• Cross-Model Review Complementarity — the review-side counterpart. Song's cross-model reviewers raise "model bias" false positives that come from their own training, and Arena's OpenAI judges show the scoring-side version: a lab taste that disagrees with both people and other labs' judges
• Screen-Relative AI Timing — a ten-model judge panel that tracked the gross effect and missed what human experts saw. That page names AI-derived text share as a self-preference channel; Arena measures the channel directly
• Reference-Free Judge Over-Crediting — the other systematic over-credit, which a reference answer corrects. Self-preference is a second, author-dependent over-credit that a reference would not obviously remove, and that page's lineage open question now has a population-level magnitude
• Version-Dependent Judge Error — judge error that moves with the agent version graded. The heatmap shows the same thing across authors at a single point in time: a fixed judge's over-credit ranges from −42 to +55pp depending on who wrote the answer
• Release Certification under Evaluator Drift — its open question on whether certification survives a judge model change. Arena gives the size of a judge-model swap on a fixed item set: up to ~50pp on a single author's answers (Astra column: +55 under Astra, +3 under DeepSeek V4 Flash)
• GPT-6 Astra, Claude Opus 5, Kimi (Moonshot AI), GLM (Z.AI), OpenAI — the judges with the strongest and weakest self-preference, and the lab with the strongest family loyalty
• Annotator as Subject: Can Gemini 3 Flash Bias Its Own Capability Profile? — this page's per-judge numbers used as a prior for a task with no own output to prefer: an LLM rating the demand of items it is then profiled on. Gemini's near-zero self-preference and OpenAI's +37pp family loyalty point the lineage concern at the GPT-4o → GPT-4o-mini overlap more than at Gemini 3 Flash

Open Questions#

• Is "AI judges agree with each other, not with people" a taste gap or a tie-scoring artifact? People pick a winner in 68% of battles, and the post does not say how a human tie scores in the agreement analysis. Recomputing Fig 5 on the battles where the human picked a winner, and reporting human–human agreement on repeat-voted battles as the ceiling, would settle it. If AI–human agreement on decisive human votes approaches AI–AI agreement, the "shared taste" finding is mostly decisiveness.
• Is self-preference self-recognition or genuine quality taste? Names were hidden, so any self-preference runs through style. Astra's +55 and MiniMax's −21.8 could both be accurate quality judgments that crowd voters miss, or recognition effects in opposite directions. The discriminating test: have a different model paraphrase each judge's own answer (preserving content, changing style) and re-judge. Self-preference that survives paraphrase is about content; one that vanishes was self-recognition.
• Does a judge-built leaderboard reorder the top of the Arena ranking depending on the judge's lab? The heatmap gives over-credit relative to humans per author, not head-to-head win rates. Computing Elo over the same 1,460 battles from OpenAI judges alone, from non-OpenAI judges alone, and from humans would show whether the family loyalty measured here is large enough to swap adjacent ranks, which is the decision a judge-built leaderboard feeds.

Sources#

• Self-Preference of AI Judges — Arena.ai, Self-Preference of AI Judges, blog post, 2026-09-30 (empirical, tier kept; COI: Arena sells human-preference evaluation). 12 judge models, 1,460 English Text Arena battles (1 Jun–23 Sep 2026), each judged twice with order swapped, 34,580 valid of 35,040 planned verdicts. Used: all body text, the four prescriptions, Methodology, and the five figures as transcribed at ingest and checked against the images for Fig 2 and Fig 3. Figure notes: Fig 1's unlabelled judge segments (opponent, tie, both-bad for several models) are bar-length readings marked ~ in the transcription and are cited only as approximate. Fig 3's diagonal differs from Fig 2 by up to ~4pp (e.g. Opus +23 vs +24.1, Kimi +25 vs +23.1, DeepSeek V4 Pro −8 vs −3.9) with no explanation on the page; Fig 2 and text values are cited for self-preference, and Fig 3 only for off-diagonal patterns. The decisive-share reconstruction of Fig 2 from Fig 1 and the +16.0pp Fig 2 mean are wiki arithmetic.