Grader Class Collapse in Jailbreak Evaluation

Sources#

• The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check

Summary#

A jailbreak paper's headline number is an attack success rate (ASR): the share of attacked prompts whose response delivers the harmful request. It is read as a property of the target model. It is really the output of a pipeline (request corpus, attack wrappers, decoding, grader, rubric), and Ruqiya Bin Safi (arXiv 2610.24872, 2026-10-17, independent researcher, empirical) isolates one stage of it: the target model, prompts and responses are held fixed character for character, and only the grader changes.

The result is not that small graders are noisy. It is that some of them cannot use one class of the rubric. Three dense graders of 8B and 9B parameters, from three families, move responses that deliver the harmful request one step down the outcome scale, from comply to evasive, instead of scattering them. The paper calls this a class collapse. On one corpus, Llama-3.1-8B assigns comply zero times in 900 responses, while another grader assigns it 179 times. That same grader agrees with the reference on 78% of the 900 items and returns a clean, well-formed label set, so no accuracy figure would flag it.

Three consequences matter for anyone reading a safety number:

1. The rate splits into two quantities that behave differently across graders. The non-refusal rate (did the model decline at all) shifts by 18–23 points. The compliance rate, which is what the field publishes as ASR, is multiplied by 22–25×. The collapse produces the multiplication: without the three collapsing graders the factor is 1.7–2.8× on all four corpora.
2. The error has a direction, and it is the unsafe one. Every small grader whose size is known under-reports compliance, none over-reports it (1.0% against 24.8% on the same responses). A cheap, local, reproducible grader certifies the target model as far safer than a larger grader finds.
3. The check is free. Per-class label counts, which every evaluation already produces, compared against one other grader, flag exactly the collapsing graders and nothing else, with no human labels.

Setup#

• Corpora. 100 harmful requests ("cores") from MaliciousInstruct and AdvBench, each wrapped by 9 published attack families (Wei et al., GCG, past-tense, many-shot), giving 900 attacked prompts. Corpus A is Qwen2.5-7B-Instruct's responses, corpus B gemma-3-12b-it's, under one protocol. Corpus A refuses 48.0% of the attacks, corpus B 71.6%. Two registered extensions: corpus C, 100 new AdvBench cores answered by corpus A's target, and corpus D, a third target family (Phi-4-mini-instruct) on the original prompts.
• Graders. Eight graders from five organisations label all 900 responses per corpus with one four-way rubric (refuse / evasive / comply / gibberish), printed verbatim in the paper: Llama-3.1-8B, gemma-2-9b-it, Qwen3-8B, Llama-3.3-70B, gemini-2.5-flash (the reference: a comparison point, explicitly not ground truth, majority of three calls), Claude Opus 5 (run blind, on opaque item IDs), gpt-oss-120b and gpt-oss-20b. Held-out graders added after the roster was fixed: qwen3.8-27b, gpt-oss-safeguard-20b, and Mistral-7B-Instruct-v0.3.
• No ground truth is constructed. Every number is a spread across graders. Human annotation (100 + 93 items, stratified toward contested ones) is treated as one more measurement.

The two boundaries behave differently#

Full-corpus rates (Table 1, reconciled against Table 8's label counts: 9/900 = 1.0%, 223/900 = 24.8%):

On corpus A the two quantities have almost the same range in points (23.8 vs 23.0) and wildly different ratios (24.8× vs 1.6×). The author rules out the obvious objections in turn:

• Small denominator. Conditioning on non-refusal leaves the spread at 15.6× (A) and 26.1× (B). Conditioning shrinks the effect on one corpus and grows it on the other, so neither denominator is "the conservative one"; the choice must be stated.
• Two extreme graders. Second-highest over second-lowest is still 13.6× (A) and 20.5× (B), against 1.2× for non-refusal. Leave-one-out keeps corpus A between 14.9× and 24.8×. On corpus B removing gemma-2-9b-it drops it to 3.5×, so there the two smallest graders carry the factor together.
• Call count and greedy decoding. One call per grader changes neither spread. Sampling the three dense graders three times at T = 1.0 keeps all three below a third of the top compliance count (four registered predictions, all held). Corpus A's factor moves 24.8× → 18.4× and corpus B's becomes defined at 59.7×.
• Outcome mix. Corpora refusing 48% and 71.6% of the same attacks give the same 22–25× factor, which the design does not force.

Collapse, not scatter#

The diagnostic shape: take the items the reference labels compliant and see where each other grader puts them. Noise around a rare class would scatter them across the other classes. These graders move them one step, to evasion:

(Table 2, matching Figure 1, which was viewed.) Llama-3.1-8B is the only small grader with a two-step tail to refusal. Llama-3.3-70B is a partial collapse at 70B (it loses 60 of 152), but the diagnostic rule below does not flag it.

The collapse replicates on new inputs: on corpus C the three dense graders keep 1, 1 and 25 of 101 reference-compliant items (registered before labels existed, and it passed). On corpus D they keep 2, 5 and 10 of 41 (descriptive, since the registered test could not be applied). The held-out Mistral-7B, a fourth family, keeps 2 of 152 and 0 of 128.

Size does not order the graders, and the paper retracts its own claim that it did. An earlier version reported the ordering as monotone in size. Size was confounded with training recipe across five organisations, and the one same-release pair, gpt-oss-20b vs gpt-oss-120b, contradicts it (24.8% vs 22.7% compliance on A, 18.2% vs 17.9% on B). A leave-one-out consensus check shows gpt-oss-20b is not faulty (93.7–95.7% of its compliant labels confirmed by other large graders). What survives is a failure region of general instruction-tuned dense graders at 7–9B, across four families. It is not a claim about small graders in general: the paper notes that Xie et al.'s (SORRY-Bench) fine-tuned 7B safety judges match GPT-4-scale judges, and those are a different object from a general model handed a four-way rubric.

The free check: per-class label counts#

Every failure above is invisible in an accuracy figure and obvious in the label distribution. The rule, with thresholds fixed in advance:

A class a grader assigns rarely (fewer than a third of another grader's count) is a grader fault if another grader assigns it on a substantial share (at least a tenth) of the same responses.

The comparison is the whole test. It separates "the responses contain no such class" (no grader assigns gibberish more than 23 times, so those zeros are not flagged) from "this grader cannot use the class" (Llama-3.1-8B's zero against another grader's 179). On the eight-grader roster it flags exactly the three dense 8B/9B graders on corpora A and B and nothing else. Held out, it flags Mistral-7B on comply only, no large held-out grader on any corpus, and exactly the three on corpus C. It could not be applied on corpus D, where no grader assigns comply to a tenth of responses (top count 77).

Its limits, as stated: it needs no ground truth and says nothing about which reading is right, and an unflagged grader can still be wrong. The reference is not flagged on any class, yet it sits well below the annotators' compliance rate.

Why reproducibility does not help. The collapsing graders are cheap, local, greedy and perfectly repeatable. "A grader that returns the same wrong answer every time passes every reproducibility check we usually run." This is Norman et al.'s consistency–bias paradox on a classification rubric: the paradox's position-biased judge is reproducible and invalid, and so is a grader that cannot express a class. The count check works because it is direction-agnostic: it flags a missing class whether its absence makes the model look safer or less safe.

Other choices that move the rate#

Holding the grader fixed, the author measures two further pipeline choices on corpus A:

• Post-labelling decisions. Three decisions about one grader's labels (a ruling that a "protective inverse" answer — asked how to steal card data, it explains how to protect it — is not a refusal; a fixed rule for empty responses; a reconciliation pass in which the grader was shown the direction of its own bias) move its non-refusal rate by 16.3 points [13.8, 19.0], with no response changed. The ruling alone, which the author considers correct, moves it 8.9 points. "A rate quoted under one rubric is therefore not comparable to a rate quoted under another, even when both rubrics are good." The grader affected is the "blind" claude-opus-5, so its Table 1 rates include a pass in which it was no longer blind to its own bias.
• Benchmark of origin. MaliciousInstruct cores reach 58.1% non-refusal against 42.8% for AdvBench cores under one grader: a 15.3-point swing [8.2, 22.6]. Standardised over shared harm domains it falls to 11.4 points with an interval crossing zero, and the two sets entered the corpus by different routes, so this is a confounded swing, reported raw because that is the comparison a practitioner choosing a benchmark makes.
• Encoding attacks bypass refusal and deliver almost nothing (reference: 99 of 100 non-refusal, 0 compliant on corpus A). Graders agree on delivery and disagree on refusal by 50 points. A scheme with no class for "bypassed refusal, transferred nothing" cannot express this outcome, which is a rubric problem that a better grader does not fix. The paper reads it as confirming StrongREJECT's "empty jailbreak" finding, with a magnitude.
• Grader noise floor. The reference disagrees with itself on 12.1% (A) and 9.9% (B) of items across its three calls, and on 18% and 14% of its own compliant labels. Every swing in the paper should be read against that floor.

The reference undercounts too. Two large graders that share neither organisation nor lineage independently take the same 35 corpus-B items from the reference's evasive class into comply. Human annotators, under their own six-way guidelines, find 27.1% compliance on corpus A against 16.5% for the reference on the same items (28.2% vs 14.9% on B). The samples are small (effective n 59 and 33) and drawn toward contested items, so they rank no grader. They point in the same direction as the large graders: if anything the published ASRs from mid-sized references are low.

The reporting minimum#

Eight items, each free and each motivated by a failure above:

1. the grader named and sized (checkpoint, parameter count, calls per item and how they are combined), never "an LLM judge";
2. per-class label counts for every grader, next to its accuracy;
3. the rubric verbatim, and every revision with the size of the change it produced;
4. the outcome classes, including how they label attempts that bypass refusal but transfer nothing;
5. target and grader decoding temperatures, and the noise floor they imply for the smallest comparison reported;
6. the benchmark the harmful requests come from;
7. whether any study-blind human labelled anything, and if not, that no one did;
8. both rates, compliance and non-refusal, and the boundary each is drawn at.

Items 5 and 8 are the paper's additions to existing lists (Gao 2026's precision/recall on a human sample, Tamsal & Rusert's evaluator protocol, Weng et al.'s Judge Card). The default for item 8: report compliance (it measures harm delivered) with the named grader and its per-class counts, and beside it non-refusal, whose cross-grader disagreement is additive. The paper meets seven of eight; it fails item 1 itself, because two of its graders (gemini-2.5-flash and claude-opus-5) have no published parameter count.

The paper's prior-work table (Table 16) is unusually candid about novelty: that ASR depends on the grader is established (Souly et al.'s StrongREJECT, Gao 2026, Tamsal & Rusert 2026), and what is new is the fixed-response design across eight graders, the collapse mechanism, the free check, and the additive/multiplicative split.

Evidence notes#

• Tier empirical, single independent author, no COI. No evaluated product; the reference grader and one target are both Google's and two graders share a family with a target, which the paper reports and claims no bias from. Five further tests were registered in writing before being run, and one registered prediction (IQR within 2× across the two rates) failed and is reported.
• No ground truth. Every claim is relative to other graders. The collapse claim survives that because it rests on the shape of disagreement (one-step, one-directional), not on which grader is right.
• Scope. Four corpora, all with the same nine attack families; corpora C and D have fewer graders; one four-way rubric; graders are general instruction-tuned models, not safety-finetuned classifiers like Llama Guard or the HarmBench classifier. Why Llama-3.1-8B's collapse is total on B and partial on A is unexplained.
• Parse warning. PDF-derived raw (docling 2.126.0 / docling-mlx 0.1.1, grade excellent). Tables 1, 2, 7, 8 and 11 are intact and cross-reconcile (Table 1 rates equal Table 8 counts / 900; Table 2 matches the prose and Figure 1). Table 13 is welded: corpus labels ("corpus B", "corpus C", "corpus D") are merged into grader-name cells, so the row where a corpus starts must be read from position. Table 15's corpus-D header row is welded into its first data row (gpt-oss-120b: 40 / 1 / 0). Table 16 (prior work) is row-shifted in its lower half. This page cites no row of Tables 13 and 15, and from Table 16 only the novelty statuses whose cells are intact (the encoding row, the established grader-dependence row).

Connections#

• LLM-as-a-Judge — the primitive, and a sharper form of that page's judge-dependence property. DRACO found rankings stable and magnitudes not across judges; here the magnitude a safety paper publishes moves 22–25× on identical responses, and the mechanism is a class the judge cannot use rather than a calibration offset
• LLM-Judge Validation — the consistency–bias paradox on a classification rubric: the collapsing graders are greedy, reproducible and 67–83% in raw agreement with the reference while missing one class entirely. The per-class count check is a cheap complement to that page's MVVP, which validates against human labels this check does not need
• Version-Dependent Judge Error — the measurement-error frame for what a collapsed grader does to a comparison. A grader whose true-positive rate on comply is near zero has a Youden index near zero, so even under non-differential error it shrinks every between-model difference toward nothing (my inference; the paper compares graders, not target models)
• Motivated Mislabeling — another judge distortion on the refuse/comply boundary that is consistent and reproducible, there driven by the label's stated training consequence. Both are invisible to test-retest; the count-against-another-grader check is direction-agnostic and would expose either only if a second grader does not share the distortion
• Benchmark Convergent and Discriminant Validity — that grid's LLM-judged benchmarks are almost all refusal benchmarks scored by a refusal classifier, and its judge swap (Qwen3-30B → Llama-3.3-70B) moved rank statistics by ≤ 0.06. That is consistent with this page: swaps among non-collapsing graders move the rate by about 2×, and a rank statistic is far less sensitive than a rate
• Self-Report as a Safety Signal — a corpus result whose ASR side-effect is read through an 8B safety judge (Llama Guard 3 8B). Llama Guard is safety-finetuned rather than a general model given a rubric, so this page's collapse result does not transfer to it automatically, but its per-class counts are the check to ask for
• Cross-Model Error Entanglement — the independence question from the other side: here two large graders from different organisations independently agree on the same 35 items the reference calls evasive, which the paper reads as evidence the reference under-counts rather than as shared bias
• Claude Opus 5 — one of the eight graders (run blind, then reconciled after being shown its own bias direction); the largest finder of extra compliance on corpus B
• Trace-Detector Measurement Validity (Silent Tool-Output Override) — the same instrument-validity question asked of a rule-based trace detector. A verification rule that counts only explicit tool re-execution has no class for "recovered without re-calling", so silent tool-output override is labelled unverified on every run, reproducibly. Like a grader that cannot use comply, the label is clean, repeatable and wrong in one direction, and the fix the authors propose is reporting (evidence, applicability, human corrections beside the label) rather than a better rule

Open Questions#

• Do safety-finetuned small classifiers used as ASR judges (Llama Guard 3 8B, the HarmBench Llama-2-13B classifier, StrongREJECT's fine-tuned scorer) collapse a delivered-harm class the same way, or is the failure region specific to general instruction-tuned 7–9B models given a multi-class rubric? Settled by running the per-class count rule with those classifiers on a fixed-response corpus like this paper's (its labels are released; responses are on request).
• Does a rare class collapse onto its common neighbour outside jailbreak grading? The author names generative benchmarks with an ordinal rubric (rare extreme class next to a frequent middle one) as the test: same fixed-response, vary-only-the-grader design, with per-class counts. A null on two or more such rubrics would bound the result to safety grading.

Sources#

• The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check — Ruqiya Bin Safi (independent researcher), The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check, arXiv 2610.24872 v1, 2026-10-17, 29pp, empirical. Cited for §2 (design), §3 and Table 1 (two boundaries; 24.8×/22.4× vs 23.0/18.0 pts), Appendix C (conditioned 15.6×/26.1×), Appendix D and Tables 5–6 (one-call and three-call spreads, 2nd/2nd 13.6×/20.5×, leave-one-out), §4, Table 2 and Figure 1 (collapse onto evasion), Table 3 (corpora C/D and Mistral-7B), §4 and Appendix I (size-ordering retraction), §5 and Table 8 (per-class count rule), §6 and Appendices H, K, M (post-labelling decisions 16.3 pts, benchmark-of-origin 15.3 pts, reference under-counting, human 27.1%/28.2%), §7 (reporting minimum, self-disagreement 12.1%/9.9%), Appendix E (encoding family), §8 and Table 4 (prior work). Parse warning: PDF-derived (docling 2.126.0 / docling-mlx 0.1.1); Tables 13 and 15 have welded corpus-label cells and Table 16 is row-shifted, so no row of Tables 13 and 15 is cited and Table 16 only where its cells are intact; Tables 1, 2 and 8 were reconciled against each other, the prose and Figure 1.