Accessibility Profiles (Breadth vs. Reliability Across Rewordings)

Sources#

• Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

Summary#

A benchmark asks each question in one wording. Users phrase the same request many ways, and models are known to answer a reworded question differently (Scale-Dependent Prompt Sensitivity). Kazem Faghih, Yize Cheng, Shoumik Saha, Mobina Pournemat, Armin Gerami and Soheil Feizi (University of Maryland, arXiv 2610.24957, 2026-10-18, empirical) make the underlying question, which they call an inquiry, the unit of evaluation. A model's accessibility on an inquiry is the probability that it succeeds on a randomly drawn meaning-preserving formulation of it. The spread of accessibility across a benchmark's inquiries is the model's accessibility profile.

Earlier multi-prompt work reports the best, worst or average score across wordings. Those summaries establish that wording matters and then collapse the shape. The profile keeps it. It separates a model that succeeds on many questions only under some wordings from one that succeeds on fewer questions under every wording, and those two can have the same mean.

Four findings carry the page.

1. Accuracy is one point inside a wide envelope. Every inquiry is reliable (right under every tested wording), brittle (right under some) or absent (right under none). Accuracy always lies between the reliable share and the latent share. The brittle share between them is 55% on TruthfulQA, 37% on MMLU and 20–24% on GSM8K, MATH-500 and BFCL at the median model.
2. Near-tied models reverse between "retry until one works" and "must work every time". Among 118 model pairs within 2 points of mean accessibility, Access@5 and Reliable@5 put the two models in opposite orders for 62 (53%). The largest gap: LLaMA-3 8B against LLaMA-3.1 8B on MMLU, +10.0 Reliable@5 and −9.3 Access@5 at mean accessibility within 0.1 points.
3. Rewording is a larger source of variation than resampling. Two answers to different wordings disagree in correctness more often than two samples of one wording in 11 of 12 model-benchmark settings. Spread over five wordings instead of five samples, the same five-attempt budget raises Access@5 by 1.1 to 9.7 points and lowers Reliable@5 by 0.5 to 11.3.
4. Model changes redistribute capability under a near-flat score. Native thinking in Qwen3 changes MMLU accuracy by −3.0 points on average, yet makes 25% of inquiries more accessible and 32% less, and cuts the reliable share by 10.9 points.

The quantities#

For inquiry q with accessibility A_q (estimated as the success share over m validated formulations), a benchmark of inquiries yields:

These are the same three functionals of a per-item success probability that Every-of-k Reliability reads as pass@1, pass@k and pass^k. The difference is the source of the repeats: there they are decoding samples of one fixed task, here they are different wordings of one question. Both collapse to µ at k = 1, and at k = 2 the trade is exact: Access@2 + Reliable@2 = 2µ.

Proposition 1 turns the "which model is better" question into a check on curve shape. If two profiles have equal mean and their reliability curves cross once, the profile that is lower before the crossing and higher after it wins every Reliable@k and loses every Access@k, for all k ≥ 1. The proof is a convex-order argument (Karlin–Novikoff cut criterion). In the data, 10 of the 11 pairs whose two gaps both exceed 2 points have curves that cross exactly once.

An appendix gives a continuous version. µ factors as effective breadth B (an inverse Simpson index of where accessibility mass sits, bounded above by L) times repeatability D (the probability a success is repeated under a fresh wording, D = Reliable@2 / µ). Changes decompose additively in logs.

Design#

• Benchmarks: SimpleQA, TruthfulQA (factual QA), GSM8K, MATH-500 (math), MMLU (knowledge), BFCL (tool calling).
• Models: 18–21 non-thinking models per benchmark from Qwen3 (0.6B–32B), Qwen2.5, Gemma-3, LLaMA-3/3.1 8B, Mistral 7B, Phi-1.5 and OpenAI (GPT-3.5, GPT-4o, GPT-4.1 / Mini / Nano, GPT-5.4-Mini), plus Qwen3 native-thinking variants for the model-change study. No Anthropic or Google Gemini model, and the largest open model is 32B.
• Formulations: generated by GPT-4.1-mini under structural cues (reorder clauses, voice, statement vs question) and stylistic cues (formal, casual, polite …). Candidates with more than 80% word overlap with the original are dropped. Survivors must pass both semantic-equivalence judges (GPT-5 and GPT-5.4-mini, so the generator does not grade itself) and bidirectional NLI entailment above 0.75 (bart-large-mnli). Five annotators labeled 1,072 pairs on SimpleQA, TruthfulQA and GSM8K. Their labels match the pipeline on 96.9–98.0% of pairs, which is human–pipeline agreement, not inter-annotator agreement.
• Retention: inquiries with at least five valid formulations, 106 (MATH-500) to 902 (GSM8K) per benchmark, median 7–10 formulations each. Every model sees the same inquiries and formulations.
• Scoring: greedy decoding with a fixed configuration. Binary benchmark-specific scorers (symbolic math verification, MMLU answer extraction, multi-judge grading for open-ended QA, structured tool-call checks). Bootstrap 95% CIs throughout.

RQ1: what the profile shows that accuracy hides#

The canonical outcome is not representative. On GSM8K the median model answers 64% of its canonical failures correctly under at least one other wording, and fails 18% of its canonical successes under at least one other wording. For a median of 15.3% of TruthfulQA and 11.3% of MMLU inquiries (up to 23.6% for one model on GSM8K), the canonical label sits on the minority side: the benchmark records a success that most wordings fail, or a failure that most wordings pass. Both directions occur on every benchmark, so the canonical wording neither favors nor penalizes models systematically. Any per-item error analysis inherits this error.

Near-ties reverse. On GSM8K four models within 2.8 points of accuracy have nearly identical latent shares and reliable shares up to 11.1 points apart (CI 8.6–13.7). On BFCL, GPT-5.4-Mini and Qwen-3 1.7B have nearly equal means and crossing curves. GPT-5.4-Mini leads Reliable@5 by 9.6 points (CI 5.5–14.0) and trails Access@5 by 6.8 (CI −10.8 to −2.8). Qwen-3 1.7B needs 1.92 attempts per solvable inquiry to GPT-5.4-Mini's 1.50, about 28% more calls, in exchange for reaching more inquiries. The expected number of retries depends on E[1/A_q], so it is set by the whole profile, not by µ.

Accuracy tracks reliability across the field and not at the resolution models are compared at. Across all models, accuracy and reliable share rank similarly (Kendall's τ 0.83–0.93 per benchmark). Among the 164 pairs within 3 points of accuracy, the reliable shares differ by more than the accuracies in 143 (87%), with a median gap of 3.8 points. In 40 pairs (24%) the more accurate model has the smaller reliable share, 16 of them significantly. This is the same scope split Every-of-k Reliability found on agent tasks, where pass@1 and pass^20 correlate across a whole field and come apart among frontier models.

Stable rankings are not evidence of equivalent behavior. Accuracy and mean accessibility order models almost identically (τ 0.90–0.99). The reliable and latent shares agree only at τ 0.56–0.85. Prior work that found rewording leaves coarse rankings intact (Lunardi et al. 2025) is reproduced here and does not imply the models behave alike.

RQ2: rewording vs. decoding noise#

The profile would only be repackaged sampling noise if a new wording changed behavior no more than a new sample does. A controlled sub-study tests that: four models (GPT-4.1-Mini, GPT-4o, Qwen-3 8B, LLaMA-3.1 8B), 150 SimpleQA, 102 TruthfulQA and 150 GSM8K inquiries, seven formulations each, five samples per formulation at T = 0.7.

• Different wordings disagree more than repeated samples in 11 of 12 settings (CI of the difference excludes zero). On TruthfulQA, Qwen-3 8B disagrees with itself in 6.6% of repeated samples and in 17.2% of answers to different wordings. The exception is LLaMA-3.1 8B on SimpleQA, which answers about 1% of inquiries correctly.
• Five wordings are both a more generous and a stricter test than five samples. Access@5 rises by 1.1–9.7 points and Reliable@5 falls by 0.5–11.3, both significant in 11 of 12 settings. The paper's conclusion: repeatedly sampling one prompt, as pass@k and pass^k usually do, underestimates the capability a model can reach and overestimates how reliably it delivers to users who rephrase.
• Greedy is not deterministic either. Repeated greedy runs of one wording differ in correctness in 0.2–4.9% of cases (a separate 200-inquiry sample; the GSM8K figure used a 256-token limit that may inflate it), against up to 16.4% across wordings. A full GSM8K greedy rerun of four models (902 inquiries, 6,596 formulations) flips 1.2–2.3% of formulation outcomes, evenly in both directions with no accuracy shift. The generated text itself changes, which points to inference-backend nondeterminism. That rerun moves the estimated accessibility of 5.5–11.6% of inquiries, which is the noise floor for the per-inquiry movement counts in RQ3. The paper measures this floor on GSM8K only.

RQ3: model changes redistribute rather than add#

119 before/after comparisons, each on one benchmark: 48 scaling steps (Qwen3, Gemma-3, GPT-4.1), 30 switches to native thinking (Qwen3), 21 switches to prompted chain-of-thought (TruthfulQA), 20 prompt optimizations (GEPA, MIPROv2 on GSM8K and TruthfulQA). The last two are sub-studies on separately generated formulation sets.

• Accessibility moves far more often than labels. On GSM8K, scaling changes the accessibility of 29% of inquiries on average (24% up, 5% down) and native thinking 27% (20% up, 7% down). Both are well above the 5.5–11.6% greedy-rerun floor measured on the same benchmark. That comparison is computed on this page; the paper reports the two numbers in separate sections. Among the 83 changes that raise accuracy, the losses amount to a median 32% of the gain. Among the 19 changes whose accuracy moves by at most one point, a median 18% of inquiries still change accessibility (up to 74%). The 18% median is within reach of the GSM8K rerun floor, so part of that movement may be backend noise.
• The same accuracy change can mean opposite things for reliability. On GSM8K the reliable share grows faster than accuracy for all ten scaling steps and all six native-thinking comparisons. On MMLU native thinking does the opposite for all six Qwen3 sizes. Qwen3-8B gains 6.2 points of accuracy and loses 4.4 of reliable share. Qwen3-32B loses 7.1 points of accuracy and 23.2 of reliable share. Most changes gain reliable share faster than accuracy on GSM8K, MATH-500 and BFCL (20 of 26, 11 of 16, 10 of 15), and 10 of 15 do the reverse on MMLU.
• Thinking on MMLU (Table 5, six Qwen3 sizes): accuracy 70.1 → 67.1, reliable share 51.6 → 40.6, latent share 86.3 → 89.5, Reliable@5 54.7 → 44.9, Access@5 84.4 → 87.2. More inquiries are reachable and fewer are dependable. In the B·D decomposition, thinking lowers repeatability for all six sizes and raises effective breadth for three, and the same broader-but-less-repeatable pattern appears for three of six sizes on BFCL. Thinking on GSM8K, MATH-500 and TruthfulQA raises both.
• Prompt optimization on GSM8K loses on every axis. Accuracy falls by 2.6 points on average (GEPA: 3.8) and 18% of inquiries become less accessible (35% for Gemma-3 4B under GEPA). Reliable share falls 60.0 → 57.0 (Table 5). On TruthfulQA the same optimizers raise both breadth and repeatability in 8 of 10 changes. Prompted CoT on TruthfulQA spreads across all four breadth/repeatability quadrants (8, 2, 5 and 6 of 21).

The paper's recommendation for model updates is to report the gains and losses behind a net change and to track per-inquiry regressions, extending negative-flip counting from single outputs to accessibility.

What evaluations should report, and the cost#

Report reliable share R and latent share L next to accuracy, and the reliability curve when two models are close. Pick the column by deployment regime. When failures are cheap to detect and retry (a verifier, test suite or human checks the output), Access@k is the relevant number. When failures can go unnoticed or feed later components automatically, Reliable@k is. This is the same split Every-of-k Reliability draws between a system with a verifier that can pick the good run and a single-run deployment.

It need not cost more than one run. Evaluating two formulations on half the inquiries, the cost of one conventional run, recovers mean accessibility as precisely as accuracy on every benchmark except the 106-inquiry MATH-500. On the four benchmarks with 600 or more inquiries it also recovers the two-formulation latent and reliable shares for every model. The reliable share is the hardest quantity to estimate when fewer inquiries remain. (These claims are quoted from the prose; Table 6, which backs them, is mis-parsed in the raw. See Sources.)

Limits#

• Formulations are LLM paraphrases, not user phrasings. They approximate but do not sample how people actually ask. Weighting by observed frequency is named as future work.
• Endpoint labels are empirical. With more formulations an inquiry can only move from reliable or absent to brittle, so the brittle shares are lower bounds.
• Small and mid-size models, single-turn tasks. The roster stops at 32B open models and GPT-4.1 / GPT-5.4-Mini; no Claude or Gemini model and no multi-turn agent benchmark. BFCL is single-call tool use. Whether the 53% reversal rate holds among frontier models on agent tasks is untested.
• Sub-studies are small. RQ2 is four models on 102–150 inquiries; prompted CoT and prompt optimization use separate formulation sets. English only, binary scoring.
• Backend nondeterminism is quantified for GSM8K only. On the other benchmarks, some per-inquiry movement attributed to wording or to a model change may be rerun noise.
• Scope of the headline. The abstract's "six benchmarks" covers the whole study. The 118-pair reversal count excludes SimpleQA, where most inquiries are absent for every model and near-tied models share almost no reliable mass.

Connections#

• Every-of-k Reliability — the same three functionals of per-item success probability with decoding samples as the repeats. This paper shows rewording moves Access@5 up by as much as 9.7 points and Reliable@5 down by as much as 11.3 against five samples of one prompt, so a pass^k number measured on a fixed wording is an optimistic bound for users who rephrase. Its near-tie reversal is ThinkingBox's GPT-6 Astra vs Kimi-K3 pattern on single-turn benchmarks
• Scale-Dependent Prompt Sensitivity — prompt sensitivity treated as a lever to tune there and as a measurable property of capability here. Both find more generation hurting on MMLU: brevity helps large models on MMLU-STEM in Hakim's study, and native thinking cuts Qwen3's MMLU reliable share by 10.9 points here
• Harness Swap vs. Rerun Noise (What a Harness Buys) — the same discipline of reading an intervention against a rerun. There a harness swap flips as many SWE-bench tasks as a rerun; here a new wording flips more answers than a new sample in 11 of 12 settings, so rewording clears the noise that a harness swap does not
• Item Response Theory for LLM Benchmarks — both re-score items rather than totals. IRT asks which items a model passes and treats each item as a fixed stimulus; accessibility asks how close to 1 each item's success probability is across wordings of the same item, and finds the canonical label on the minority side for 11–15% of MMLU and TruthfulQA items
• Benchmark Score Redundancy — aggregate rankings that hold steady while behavior differs. Accuracy and mean accessibility order models at Kendall's τ 0.90–0.99 while reliable and latent shares agree at 0.56–0.85, a reminder that a low-rank, rank-stable score matrix says nothing about how dependable each score is
• Measuring Beyond Accuracy Saturation — the reliability axis taken from run-to-run consistency to wording-to-wording consistency, on unsaturated benchmarks
• Agent-Authored Harness Optimization — a further data point on prompt optimization as the weakest self-improvement axis: GEPA lowers GSM8K accuracy by 3.8 points and makes up to 35% of inquiries less accessible for one model

Open Questions#

• Does the near-tie reversal rate between Access@5 and Reliable@5 (53% of pairs within 2 points of mean accessibility) hold among frontier models on multi-turn agent tasks? The roster stops at 32B open models and GPT-5.4-Mini. Settler: accessibility profiles from paraphrased task statements on τ²-bench or SWE-bench Verified for Claude, GPT and Gemini frontier models, with a rerun floor per model.
• Are brittle shares and model orderings the same when formulations are drawn from real user phrasings rather than LLM paraphrase? Settler: the same models profiled on intent-clustered rewordings from a production query log, frequency-weighted, compared with the paper's paraphrase-based profiles on matched inquiries.

Sources#

• Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles — Kazem Faghih, Yize Cheng, Shoumik Saha, Mobina Pournemat, Armin Gerami & Soheil Feizi (University of Maryland), Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles, arXiv 2610.24957, 2026-10-18, 26pp, empirical. Code, formulation sets, model answers and judge grades released. Cited: §3 definitions and Proposition 1 (Appendix A proofs), §4 setup and Appendix B formulation pipeline with Table 1 human agreement, §5.1 and Appendices E–F (envelope, majority reversals, near-tied pairs, Kendall correlations), §5.2 and Appendix G (rewording vs resampling, greedy rerun Table 4), §5.3 and Appendix H (model changes), §6 (deployment regimes, retry budget, two-formulation estimation), Appendix K (B·D decomposition), Tables 3 and 5. Parse warning: PDF-derived (docling 2.126.0 / docling-mlx 0.1.1, grade excellent). Tables 3 and 5 were reconciled row for row against pdftotext -layout and are clean. Table 2 (model roster) prints several models per row in the PDF itself, so the parse's multi-value cells are faithful, not a collapse. Table 6 is row-shifted in the parse: SimpleQA's K = 3 row (n = 206) is welded into TruthfulQA's row (the "206 235 157" cell), and MMLU's K = 3 row (n = 211) into BFCL's. No Table 6 cell is cited here; the budget claims are quoted from the Appendix J prose. Figures were not opened: every figure number used is stated in the prose. The greedy-rerun-floor comparison in RQ3 is computed on this page from two of the paper's numbers