Usage-Telemetry Classifier Validation

Sources#

• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
• Google's AI & Economy ATLAS v1.0: Mapping Gemini Usage in the Economy

Summary#

The entire genre of AI-usage economics — Anthropic's Economic Index, OpenAI's Codex study, Google's ATLAS — rests on one unexamined layer: an LLM reading a conversation summary and assigning it to a cell in an official statistical taxonomy. Every "X% of tasks," "Y% automation," "Z% of occupations" claim inherits that classifier's error. Until ATLAS v1.0, nobody published how big it was.

ATLAS did, across three independent methods, and the headline is uncomfortable: the classifier assigns the exact O*NET task correctly 22.6% of the time, and human expert raters approve 85.8% of those same assignments as economically reasonable. Both numbers are real. The gap between them is the finding.

Evidence note. empirical — three validation studies: accuracy against 18,801 synthetic ground-truth conversations, inter-rater agreement against three in-house annotators (N≈110–120 clusters), and a human approval study on 240 clusters. Google's own caveat: synthetic data may be easier (a Gemini-generated prompt classified by a Gemini classifier may carry recoverable cues) or harder (an ambiguous prompt may genuinely fit a category better than its seeded label, penalizing a correct classifier).

The scale of the problem#

Before accuracy, the arithmetic of difficulty. ATUS has 456 detailed activity codes; O*NET-SOC has 1,016 occupational titles and 18,797 task statements. Random assignment scores 0.22%, 0.10%, and 0.005% respectively. Even at the coarsest SOC 2-digit level (23 categories), chance is 4.35%.

And the categories are genuinely ambiguous, not just numerous. "Help me understand the tax implications of this contract" is defensibly a lawyer, an accountant, or a business operations specialist. "Instructions for building a nest box with a 10-year-old" splits on whether the child is a household member — a distinction ATUS makes at the top tier and users have no reason to disclose.

Crucially, this ambiguity predates AI. Mellow & Sider (1983) found 17% of CPS respondents disagreed with their own employer about their 1-digit occupation, rising to 42% at 3-digit. Mathiowetz (1992) found 24% and 48%. The category system was never crisp; LLM classification inherits a measurement problem economics already had.

What the accuracy numbers say#

Taxonomy	Level	Categories	Random	Accuracy	Lift
ATUS	Tier 1 (Major)	17	5.88%	72.84%	12.4×
ATUS	Tier 2	108	0.93%	42.91%	46.3×
ATUS	Tier 3 (Detailed)	456	0.22%	23.70%	108×
SOC	Major Group	23	4.35%	71.57%	16.5×
SOC	Minor Group	98	1.02%	58.44%	57.3×
SOC	Occupation Title	1,016	0.10%	42.47%	432×
O*NET	Specific Tasks	18,797	0.005%	22.58%	4,244×
O*NET	Autor–Thompson type	5	20%	70.44%	3.5×

Plus the gating decision: the work/non-work classifier hits 93.7% on a balanced synthetic set — which matters more than any other single number, because an error there routes the conversation into the wrong pipeline and propagates through everything downstream.

Read the table in two directions and it tells opposite stories. Absolute accuracy collapses with granularity (71.6% → 22.6%). Discriminative lift explodes (16.5× → 4,244×). Both are true; which one you quote determines whether the pipeline sounds trustworthy.

The accuracy/approval gulf#

The second study asked three human annotators a single yes/no per label: could most activities in this cluster fit the assigned label, or is any mismatch minor enough not to distort the economic interpretation?

Work level	Approved	Non-work level	Approved
SOC Major Group	96.1%	ATUS Tier 1	98.1%
Occupation Title	92.8%	ATUS Tier 2	96.1%
Task Statement	85.8%	ATUS Tier 3	92.2%

So at the O*NET task level: 22.6% exact, 85.8% approved. Google names the divergence and does not resolve it — "the drivers of this divergence, and its implications for improving classifier validation require further investigation" — but reads the two as bounds: high approval means granular labels carry real economic meaning; low exact accuracy is "a caution against over-relying on hyper-specific task analysis."

A third study makes the same point from the agreement side. Against human plurality consensus, the SOC Major Group classifier scores Cohen's κ = 0.83 ("almost perfect"), while human-only inter-rater Fleiss κ is 0.68. Adding the LLM as a fourth rater either significantly improved agreement (work classifier, SOC Minor Group; p<0.05) or didn't shift it (all ATUS, SOC Major). The model rates about as consistently as the humans do — which is a statement about how noisy the humans are as much as how good the model is.

The tell: rater approval exceeds rater-model plurality agreement by 11–15pp at the top taxonomy levels. Presented with a label, annotators accept it; asked to pick one unprompted, they choose differently. The taxonomies are ambiguous enough that multiple answers are defensible, which means plurality-agreement metrics understate the classifier and approval metrics flatter it.

The mitigations that actually work#

ATLAS's response is methodological rather than a better classifier, and this is the transferable part:

1. Aggregate into economically meaningful groups. Exact task assignment is 22.58%; the same assignments land in the correct Autor–Thompson task type 70.44% of the time. Grouping absorbs the noise without discarding the economics — which is why Task Saturation: Broad but Shallow AI Diffusion's task-type analysis is more defensible than any individual task claim.
2. Measure presence, not frequency. Whether a task was observed at all is far more robust to misclassification than how many times. This is why saturation thresholds (25 users, 50 users) rather than interaction counts drive the headline numbers.
3. Merge categories the taxonomy splits for non-economic reasons. Rolling up travel-by-purpose codes and merging household/non-household pairs lifts ATUS Tier 2 accuracy from 42.9% to 51.7% and Tier 3 from 23.7% to 32.4% — nearly +9pp each, purely from not distinguishing things users have no reason to disclose.
4. Randomize option order in the classifier prompt, to defeat documented LLM position bias.

Errors are structured, not random#

The confusion analysis (normalized to excess lift over base rate) shows misclassifications concentrating in semantically adjacent categories rather than scattering: Management ↔ Business & Financial Operations at 2–3× base rate, Sales ↔ Office & Administrative Support, and in the non-work domain, caregiving errors clustering on household vs non-household members — exactly where conversational context is silent.

Structured error is better news than random error for aggregate measurement (the distortions stay local and modest) and it is actionable: it points at category merges, prompt clarifications for specific confusable pairs, and even ex-post reweighting. But it also means errors are correlated with the economic distinctions people care about — Management vs Business Operations is not a distinction that washes out when you aggregate to "white-collar."

Why this generalizes beyond ATLAS#

Three reasons this page is about the genre, not one report:

• Everyone else's numbers rest on the same layer and none of them published this. The AEI's Clio pipeline and OpenAI's Codex classifiers do the same job with the same class of error, validated to the public only in general terms. ATLAS's contribution is not that its classifiers are good — it's that it showed its working. Cross-study comparisons (~20% vs 36% vs 49% task coverage) should be read knowing that at least one instrument has ±77pp of exact-assignment error at that level of granularity, and the others are unquantified.
• The metric choice determines the story, exactly as in LLM-Judge Validation: there, exact-match agreement overstates chance-corrected κ by 33–41pp; here, approval overstates exact accuracy by 63pp at the task level. Both are cases of a validation metric flattering an LLM classifier because it never asks the model to discriminate under the conditions the headline claim assumes.
• The ambiguity is in the world, not the model. The 42–48% human-human disagreement on 3-digit occupations means there is a ceiling no classifier can pass. That reframes the goal from "make the classifier accurate" to "choose aggregations coarse enough to be identified" — a constraint on what usage telemetry can ever measure, not a temporary engineering gap.

Connections#

• GDPval Benchmark — the same reporting discipline reached independently by a different lab on a different measurement problem, and the cleanest external support for this page's central argument. GDPval's automated grader (GPT-5-high, scoring professional deliverables) is validated not against a fictional perfect label but against what human expert graders achieve against each other: 65.7% grader–human agreement against a 70.8% human inter-rater ceiling, so the headline is a 5-point shortfall rather than a 34-point failure. It is the same correction this page's resolved question arrived at — score the automated rater relative to the human ceiling — applied where the "ground truth" is a preference rather than an occupation code, and it comes with the same honesty tax attached: OpenAI also publishes that both agreement metrics are highest on weak models, whose outputs are easy to tell apart from a human's, which is this page's accuracy/approval gulf in another guise
• Task Crossover — a measure that is classifier-mediated twice over: both the user's occupation and the task's "home" occupation are model judgments, so this page's error bar applies to it doubly
• Google AI & Economy ATLAS — the program that published these numbers; validation is its clearest methodological contribution
• Task Saturation: Broad but Shallow AI Diffusion — the headline finding most exposed to this error, and the reason its task-type analysis is more trustworthy than its per-task claims
• The Household Production Boundary — ATUS Tier 3 at 23.7% exact accuracy makes the granular household activity claims the widest-error-bar numbers in the report
• LLM-Judge Validation — the sibling result in the evaluation domain: same structural failure (a lenient agreement metric flattering an LLM classifier), same remedy (chance-correct, and validate under the conditions the claim assumes)
• LLM-as-a-Judge — the general pattern; a taxonomy classifier is a judge with 18,797 options instead of two
• Anthropic Economic Index — the rival program whose classifier error is unquantified in public, which is now a visible asymmetry rather than a shared silence
• Telemetry vs. Survey Measurement — telemetry beats self-report on latency and scale, but this page is the price: the telemetry is only as good as the classifier that reads it, and that classifier has never been the object of study
• Measuring Beyond Accuracy Saturation — the same move of interrogating what a headline metric can and cannot support
• Matched Comparisons for Memorization Claims — the same discipline in the memorization literature, and the cleanest statement of why it matters: an extraction rate measured only on training data has no error term until matched non-training sequences are scored the same way, at which point ~24% of the apparent rate at 10-token suffixes turns out to be predictability. ATLAS measures its instrument against human labels; Cooper et al. measure theirs against a null that cannot possibly be positive — the stronger version of the same control, available only because non-membership is definitionally exclusive
• Reference-Free Judge Over-Crediting — a judge scoring generously in the absence of ground truth; the human-approval study is structurally the same setup and produces the same optimism
• Exposure Taxonomy: Observed, Theoretical, Reported, Anticipated — every exposure measure is a classifier output; this quantifies the error bar the taxonomy's distinctions ride on
• What the Instrument Can Resolve: Two Headline Numbers and Their Missing Denominators — supplies the ceiling this page asks for and renormalizes against it: leave-one-annotator-out agreement computed on the same items (an estimator ATLAS's three-rater, N≈110–120 annotation already supports and never runs), giving 73–82% of the human ceiling at occupation title and 85–94% at major group, with O*NET tasks provably not renormalizable — which recasts category merging as raising the ceiling rather than reducing noise. Also the reason the 85.8% approval rate cannot serve as the ceiling (anchored vs blind), and a worked case of this page's error bar landing on an economics result: the 25%/7% screen-out split in Controlled Variance: AI's Edge as Reduced Dispersion is an unvalidated ten-way LLM label whose codebook thresholds on a variable the treatment moves

Open Questions#

• The synthetic ground truth is Gemini-generated and Gemini-classified. How much of the 22.6% is real capability vs same-family cues, and how much is the classifier being penalized for picking a better fit than the seed? Google says it cannot measure this without inspecting private logs.
• Would running ATLAS's validation battery against the AEI and OpenAI classifiers explain the 2–4× cross-study disagreements, or are the gaps driven by sampling and product mix instead?

Resolved Questions#

• Human raters disagree with each other on 42–48% of 3-digit occupations. Is there a principled way to establish the ceiling a classifier could reach, so accuracy can be reported relative to it rather than to 100%? Answered (2026-08-17) by What the Instrument Can Resolve: Two Headline Numbers and Their Missing Denominators: yes — leave-one-annotator-out agreement on the same items, with accuracy reported as (observed − chance) / (ceiling − chance). Hold out one rater, predict their label from the others, and score the classifier by the identical procedure — the estimator Yang et al. use to show PandaLM retains headroom (best judge κ = 0.753 against a human ceiling of 0.920) while Judge's Verdict is at or past its noisy ceiling (0.620 against 0.562). ATLAS already collected what this needs and stops one step short: B.3.1's three raters over N ≈ 110–120 give model-vs-plurality κ = 0.83 against human-only pairwise κ = 0.66 / Fleiss 0.68 for SOC Major — but agreeing with a plurality of three is an easier target than agreeing with one drawn rater, so those columns are not comparable; the held-out recomputation costs no new annotation. The sibling instrument, from LLM-Judge Validation's Shopify account, is to measure the ceiling before the classifier exists, on the actual rubric by the actual annotators, with κ ≈ 0.2 as a rubric-rewrite trigger — where ATLAS's ceiling is borrowed from Mellow & Sider (1983) and Mathiowetz (1992). Renormalizing this page's table against those borrowed ceilings: SOC Occupation Title 42.47% → 73–82% of the human ceiling, SOC Major Group 71.57% → 85–94%, both understated because the borrowed ceilings sit at coarser granularities than the levels they are applied to. Three limits are part of the answer rather than gaps in it. (i) The approval rate cannot be the ceiling — 85.8% is an anchored statistic (the label is shown) while 42–48% disagreement is a blind one, and this page's own 11–15pp approval-over-plurality gap prices the difference, with Reference-Free Judge Over-Crediting measuring its extreme at FPR 0.719 → 0.012 under commit-first de-anchoring. (ii) At O*NET-task granularity no ceiling is estimable at all, and ATLAS proves it (B.3.1): when categories vastly outnumber rated observations, p_e is overestimated, κ underestimated, and the computed value is driven by whichever categories were sampled — so 22.58% has no reportable denominator and will not acquire one by hiring more raters. (iii) Consequently aggregation is the ceiling intervention: rolling tasks into Autor–Thompson types (→ 70.44%, 5 categories, agreement study feasible) and merging distinctions users have no economic reason to disclose (Tier 2 42.9 → 51.7%, Tier 3 23.7 → 32.4%) raise ceiling and accuracy together, because the merged distinctions are exactly the ones a human rater reading the same conversation could not make either. The ceiling is a property of the category system, not of the classifier.

Sources#

• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Patwardhan et al. (19 authors, OpenAI), arXiv 2510.04374 v1, 2025-10-05, 29pp (empirical). Cited here only for the ceiling-relative reporting parallel: §2.5 and Appendix A.6.1–A.6.2 — the agreement definition E[1 − |H − A|] on ordinal {0, 0.5, 1} scores, 65.7% automated-grader agreement against a 70.8% human inter-rater ceiling (Figure 4b, viewed at compile time), and the note that both metrics run highest on weak models. Full treatment on GDPval Benchmark
• Google's AI & Economy ATLAS v1.0: Mapping Gemini Usage in the Economy — ATLAS v1.0 Appendix B: Classifier Validation (B.1 Measurement Challenges, B.2 Synthetic Accuracy, B.2.2 Misclassification Patterns, B.3 Human Comparison), Tables 5–7, Figures 30–31