Annotator as Subject: Can Gemini 3 Flash Bias Its Own Capability Profile?

Sources#

Using profiles of cognitive capability to assess AI suitability for workplace tasks

Question#

From the #oq/now bullet on Cognitive Capability Profiling for Task Suitability: Prunty et al. had GPT-4o and Gemini 3 Flash annotate the 0–5 demand level of every battery item on 18 capabilities. Gemini 3 Flash was then one of the six profiled systems. Does an annotator that is also a subject bias its own profile? The paper's circularity paragraph separates the roles ("a rubric-applying classifier rather than a subject") but never addresses this overlap.

Short answer#

Probably not by much, but nobody has measured it. Synthesis settles four things:

Only one channel matters, and it is an interaction. A bias that hits all subjects alike cannot favour Gemini 3 Flash.
The mechanism measured elsewhere is missing here. Self-preference has been measured when a judge rates its own output. Demand annotation rates third-party item text.
Every prior points toward a small effect. Gemini-family judges show the least self-preference on record. Averaging halves any per-item bias. The verbal difficulty ratings a model gives track generic difficulty, not its own success.
The published tables show no footprint.

What synthesis cannot settle is the size of the item-level interaction. That needs a refit on the released annotations, ideally with an annotator from a third family. The question is retagged #oq/source.

1. Which bias could move the subject's own profile#

The Measurement Layouts model on Cognitive Capability Profiling for Task Suitability fits a margin c_k − λD_jk per item and dimension, with one intercept shared across the catalogue. Capability levels are identified only relative to the other five systems.

A rater-wide main effect cancels. The paper says the two raters disagree "largely" through "small, capability-specific differences in scale use": GPT-4o rates Cognitive Flexibility higher, Gemini rates Semantic Memory higher (Using profiles of cognitive capability to assess AI suitability for workplace tasks §3.1). An offset like that raises the demand on every item for every subject. It moves all six systems' estimates on that dimension together. Under a shared intercept that is a scale shift, not an advantage for anyone. Gemini 3 Flash's Semantic Memory (6.12) is not even the catalogue's top value: o4-mini has 6.34.
Only a subject-coupled interaction can bias the profile. To favour itself, the annotator must rate items differently depending on whether it would answer them correctly. The pattern would be lower demand on the items it passes and higher on the items it fails, or the reverse, beyond what GPT-4o does on the same items. This is the shape Same-Model Review Blindness found in Greptile's 2×2. Its reviewer and dataset main effects were 0.6pp and 2.6pp, and the whole signal sat in the 8.9pp crossover. A family effect is an interaction, and the test must be one too.
The direction on the level is ambiguous. Rating your own passes as easier weakens the lower bound they put on c_k. Rating your own failures as harder loosens the upper bound. The two pull the level estimate in opposite directions. Their common effect is to make the subject's response pattern fit the demand matrix more cleanly. So the most likely footprint is a sharper posterior, not a higher level. The question's "bias" is therefore as much about how precise Gemini 3 Flash's profile looks as about how high it is.

2. The mechanism that has been measured does not apply#

Every self-preference number in the wiki comes from a task where the judge sees its own product:

Arena: 12 judges pick between two answers, one of them their own (+55.3pp for GPT-6 Astra down to −21.8 for MiniMax M3).
Greptile: reviewers look for bugs in code their own family wrote.
Song: reviewers check artifacts Claude wrote.

Demand annotation shows the annotator a benchmark item from Fantom, PlanBench, AGIEval and so on, which no subject wrote, with no subject response attached. Recognising your own text, the most direct route to self-preference, is absent by design. That is the paper's role-separation argument, and on this narrow point it holds.

The channel that remains is self-knowledge leaking into difficulty ratings: an annotator calling items hard because it, specifically, finds them hard. The wiki's evidence on that runs against a strong effect. On Pre-Reasoning Commitment and Confident But Unsure, a model's stated confidence tracks generic question difficulty. Its own item-level correctness is readable from activations (AUROC ≈ 0.76) but is not passed through the verbal report ("overconfidence is less a failure of self-knowledge than of self-report"). A rubric-anchored 0–5 rating is a verbal report of that kind. It should carry the generic-difficulty signal both raters share, and only a weak self-specific one.

3. Priors on the size#

Gemini judges self-prefer least. In Arena's per-judge table, Gemini 3.8 Flash scores +3.5pp with an interval that includes 0. That is one of the five judges with no detectable self-preference, against +24 to +55 for the six that show it (LLM-Judge Self-Preference). It is a later Gemini generation and a different task, so it is a prior, not a measurement. It still says the lab whose model is in question is not one of the strong self-favourers.
The family loyalty that was measured belongs to the other annotator's lab. Arena's OpenAI trio rated each other +37pp above people. That makes the OpenAI side of this design the stronger candidate for a lineage effect, if either side has one. GPT-4o is the parent family of the profiled GPT-4o-mini, and Kuai et al. measure GPT-4o ↔ GPT-4o-mini among the most error-aligned pairs (CIG 0.0523, second only to the Llama-3 pair). The question names Gemini 3 Flash, but the design has a sibling overlap on both sides. The same doubt applies to Gemini 3.1 Pro and Gemini 2.5 Flash as family members.
Averaging halves it. The final demand is the mean of the two raters. Any per-item self-coupled shift Gemini 3 Flash introduces reaches the fitted matrix at half strength. On the 16 retained dimensions the raters agree within one level on 74–97% of items (Table A2), so on most items the most Gemini can move the mean is half a level.

4. Footprint in the published tables#

If the coupling were large, Gemini 3 Flash's profile should look unusual against the others. It does not:

Level rank matches accuracy rank. Gemini 3 Flash is second on battery accuracy (68.4%) and second on overall level (3.38). The IRT model reorders only one pair: GPT-4o-mini rises above GPT-5-nano despite 7.3 points less accuracy. That reordering benefits the other annotator's sibling. The paper attributes it to GPT-4o-mini's correct answers concentrating on high-demand Language and Social Cognition items. The Social Cognition components have the best inter-rater agreement in the paper (ToM ρ 0.81, Emotion Perception 0.72), which leaves little room for one rater's idiosyncrasy. So the reordering is not evidence of an annotator effect either.
The posterior shape is typical. Under the shared intercept, Gemini 3 Flash has 4.84 effective dimensions (catalogue range 4.54–5.68), mean |r| 0.22, and condition number 25, the same as Gemini 3.1 Pro (Table C4). Its posterior SDs (0.13–0.44, Table C6) sit inside the range the other systems show. The sharper fit predicted in §1 is not visible.

This is absence of a large signal, not evidence of zero. Tables of six systems cannot bound an effect of a few tenths of a level.

What would close it, and the control the bullet's test lacks#

The bullet's experiment refits all six systems under GPT-4o's matrix alone and under Gemini 3 Flash's alone, then checks whether the Gemini 3 models move more under their own annotator. It has the right shape: a difference-in-differences, the interaction §1 says is the only channel. Three refinements follow from the evidence above:

It measures differential bias only. GPT-4o is entangled with GPT-4o-mini, and Kohli and Hossain show error structure crossing vendor lines (Cross-Model Error Entanglement: frontier cross-provider ρ 0.42 vs 0.40 within the Gemini family). If both annotators favour their own families by similar amounts, the two refits move symmetrically and the test reads null. A third-family annotator (Claude, or an open-weight model) gives the neutral baseline. A human-expert subsample on the high-disagreement dimensions (Working Memory, Mental Simulation, Cognitive Flexibility at ρ ≤ 0.40) gives the anchor.
Separate self from family from capability. The roster can already do this. If only Gemini 3 Flash moves, the effect is self. If Gemini 3.1 Pro and 2.5 Flash move with it, the effect is family. If movement tracks capability across both labs, the variable is capability. Version-Dependent Judge Error and Song both find capability structuring judge error more than lineage does.
Test the subject-coupling directly before refitting. For each annotator, compute the partial correlation between its demand rating and each subject's pass/fail on the item, controlling for the other annotator's rating. A self-coupled annotator shows a stronger partial correlation with its own pass/fail than with the other subjects'. This needs only the released annotations and response matrices, not a Bayesian refit.

The data for all three is public (github.com/Kinds-of-Intelligence-CFI/Task-Suitability-Profiles), but the wiki holds the paper, not the annotation files. Closing the question needs that analysis run, or a source that runs it.

Sources#

Cognitive Capability Profiling for Task Suitability and Using profiles of cognitive capability to assess AI suitability for workplace tasks: §3.1 (the two annotators, rater scale-use differences, averaging), the circularity paragraph, Tables A2, C4, C5 and C6.
LLM-Judge Self-Preference: per-judge self-preference (Gemini 3.8 Flash +3.5pp, interval spans 0) and OpenAI family loyalty (+37pp).
Same-Model Review Blindness: lineage bias as an interaction rather than a main effect.
Cross-Model Error Entanglement: GPT-4o ↔ GPT-4o-mini CIG 0.0523, and vendor as a weak index of shared error structure.
Pre-Reasoning Commitment, Confident But Unsure: stated confidence tracks generic difficulty, not the model's own correctness.
Version-Dependent Judge Error, Cross-Model Review Complementarity: capability as the competing variable.