Release Certification under Evaluator Drift

Sources#

Summary#

A model update ships when its risk on the current population stays under a threshold. Trusted labels (expert safety ratings, execution results) settle that but are expensive. A cheap evaluator, usually an LLM judge, scores everything. Prediction-powered evaluation splits the risk into a free part and a paid part: E[Y] = E[Q] + E[Y − Q], where only the mean residual E = Y − Q (the evaluator's average error) needs labels. Earlier audits leave a ledger of exactly that residual, so the tempting move is to reuse it: the model changed, but maybe the judge's errors did not.

Mondal, Banik, Apurba & Al Mahmud (arXiv 2609.32267, 2026-09-26, empirical, theory plus prespecified experiments) answer when that reuse is legitimate. The answer turns on the status of the history, not on its accuracy:

M0, hidden residual drift. The evaluator's error can change while every cheap observable keeps its distribution. History need not constrain the present.
M1, trusted transport. A declared radius ρ bounds how far the residual can move. History may enter validity, and a candidate is certifiable with zero fresh labels when Q̄_tar + U_H + ρ ≤ τ, at error cost δ_H + δ_T.
M2, untrusted advice. History may predict the current residual but is not assumed correct. It may set bets, never the null, the threshold or the error guarantee.

The paper's own one-line positioning: prediction-powered inference, e-values, betting confidence sequences and expert mixtures all predate it. The contribution is separating history that may enter validity from history that may only guide label collection. Its recommendation for release practice: every certificate should state which of the two roles historical evidence played.

The impossibility half: transport must be assumed, not tested#

Proposition 2 (no label-free validation of transport). Any test computed from the cheap transcript has the same law under a stationary safe world and a world where one coordinate silently became unsafe. So residual stability cannot be verified from judge scores; it can only be assumed or checked with labels. Corollary 7 closes the adaptive loophole: deciding from the same pre-label transcript whether to trust the bridge is covered by the same equality.

Theorem 1 (price of vigilance). In a Bernoulli submodel of M0 with M monitored coordinates (deployment blocks × auditable strata), any certifier that is valid across coordinates and live on a safe stream needs at least kl(1−β‖δ) / kl(q₀‖q₁) expected labels per coordinate, charged on the stationary path where nothing changes. Guarding against a possible change costs labels even when none occurs. A by-product: with probability at least 1 − β − δ, every coordinate gets at least one label. A sequential likelihood-ratio certifier attains the bound up to an overshoot constant (expected-label minimax rate Θ(M(q₁−q₀)⁻² log(1/δ))); a deterministic per-coordinate label cap pays an extra log M.

Theorem 3 (price of trusted transport). The zero-label certifiable set shrinks as ρ grows and is empty once ρ exceeds the largest slack; ρ → ∞ recovers M2. The repeated-judge data make this concrete (Fig. 7, viewed): on DICES-990 at ρ = 0 a zero-label certificate would cover 81% / 77% of candidate-block cells under rubrics A / B, but the bridge actually holds for only 35% / 50%. The radius at which the bridge first holds everywhere (0.12 / 0.10) leaves 0% / 2% certifiable. A bridge wide enough to be true removes the zero-label advantage. ToxicChat (one outcome, eight blocks) is more forgiving, 62.5% / 100% certifiable at a radius that holds, but nothing in cheap data identifies that radius in advance.

The real-data failure (Appendix L, exploratory, earlier certifier family). On Folktables ACSIncome/ACSPublicCoverage with injected hidden failures (one proxy-decile cell's trusted losses raised, proxies and bridge untouched), a certifier that trusts a calibrated bridge without verifying it false-certifies 0.417 / 0.750 of injected blocks at margins 0.038 / 0.05, block for block identical to a zero-label oracle, while every fresh-label baseline catches all 120. At the prespecified 0.02 margin the bridge (width 0.048) is wider than the margin, transports nothing, and no method fails. The larger margins were chosen after this diagnostic.

The constructive half: portfolio vigilance#

The certifier samples the target pool in one uniform random order without replacement and bets against the null R(k) ≥ τ_k after each purchased label. Wealth starts at 1, each factor has conditional mean ≤ 1 under the null, and Ville's inequality bounds the chance it ever reaches K/δ. Two experts bet:

Ledger expert: forecasts the remaining-pool mean as Q̄ + Ā (the historical residual) from the first label.
Robust expert: starts with no residual correction and learns it only from purchased target labels.

The deployed evidence is the fixed 50/50 mixture of their wealths. Theorem 4: valid for arbitrary historical advice, candidate-specific stopping, shared purchased rows and abstention at budget, because advice enters only through the information before each draw. Each assumption is needed: non-uniform (convenience) sampling, a stake that peeks at the next row's proxy, or thresholds set from target outcomes each break the supermartingale. Theorem 5: the mixture trails the better expert by at most log 2 in log-evidence, w = 1/2 uniquely minimizes the worst case, and in label units the portfolio reaches any evidence level no later than the first expert reaches twice it. Undecided candidates are reported as abstentions, "never silently safe".

What history can and cannot buy (Proposition 6, canonical model)#

With advice mismatch η = μ_tar − μ_hist, the ledger's evidence growth rate is G_L(η) = kl(p‖m) − kl(p‖p−η) (an exact identity, Theorem 10). It peaks at η = 0 at exactly the robust expert's rate kl(p‖m).

History never raises the evidence growth rate. Accurate advice only matches what the robust expert reaches by learning. Its value is a finite-sample head start.
Damage is asymmetric. Pessimistic stale advice makes the ledger stop betting and costs power. Optimistic stale advice beyond η⁺ makes its evidence decay to zero.
The portfolio keeps the vigilance rate for every η, and its price for not knowing η is at most log 2 / G_L(η) extra expected labels. At η = 0 its bound matches Theorem 1's lower bound to first order in log(1/δ).

In simulation (Fig. 2, viewed; p = 0.20, m = 0.25): exact advice gives ledger 430.3, portfolio 492.3, robust 790.6 mean labels. At η = −0.06 the ledger never certifies within 6,000 labels while the portfolio needs 913.4. At η = +0.06 the ledger fails on 38.7% of paths and the portfolio needs 751.0.

Results#

Held-out natural data (prespecified protocol, 5 seeds, block-level δ = 0.05, β = 0.10). Paired portfolio/PPRM-style label ratio at full budget: 0.465 (95% CI 0.327–0.575) on CIFAR-10N and 0.740 (0.618–0.877) on DICES-990, reductions of 53.5% and 26.0%. No false certification; one DICES false rejection, shared with the robust expert. Resolution AUC 0.880 vs 0.743 and 0.569 vs 0.402. At 20% budget the portfolio resolves 90% vs 40% (CIFAR-10N) and 34.58% vs 13.75% (DICES). On these data history is accurate, so the ledger-only expert is slightly cheaper than the portfolio (248.9 vs 279.9 labels on CIFAR-10N): the hedge has a visible price when it is not needed. The robust-only expert alone (967.8) is costlier than the PPRM-style monitor (574.6) on CIFAR-10N, so the saving comes from history used as advice, not from betting per se. DICES per-block ratios run 0.422–0.999: U.S.-locale blocks (0.42–0.85) gain far more than India-locale ones (0.86–1.00), with Spearman −0.91 against distance to threshold and +0.72 against ledger forecast error (n = 12, descriptive). All six external blocks (noisy CIFAR-10, SVHN, UCI Naval) also favor the portfolio (0.296–0.916). Threshold margins 0.02–0.10 keep the ratio at 0.403–0.465 / 0.649–0.740.

Corrupted advice (synthetic pools). Across seven advice biases the portfolio stays within 8.0% of the cheaper component; ledger-only reaches 1.66× it. At a −0.15 bias: ledger 986.7, portfolio 624.5, robust 594.5.

A real evaluator change: rubric drift in one LLM judge (post-confirmatory). One fixed model (GPT-5.6 Luna) judges every DICES-990 conversation and all 5,654 human-labelled ToxicChat-0124 examples three times under a legacy narrow safety rubric (A) and an updated contextual-risk rubric (B); A forms the ledger, B the current proxy. The rubric change is real, not judge noise: between-rubric mean absolute score change exceeds within-rubric by 0.0476 (0.0331–0.0633) on DICES and 0.0379 (0.0301–0.0460) on ToxicChat. After the change:

Portfolio/PPRM ratio 0.790 (0.667–0.909) on DICES-990 and 0.631 (0.520–0.770) on ToxicChat, no false certification.
Stale history now makes ledger-only the costlier expert: the portfolio needs 140.7 fewer labels than ledger-only on DICES (−256.2 to −46.2) and 12.7 fewer on ToxicChat (−20.8 to −5.9), cheaper in all nine A-run × B-run pairings on both datasets.
Against robust-only it wins on DICES (−109.1) and ties on ToxicChat (+0.05, −4.65 to 4.05); the authors claim no advantage there.
The DICES ratio under shift (0.790) is not better than the primary 0.740. The authors' reading: the hedge protects the portfolio; it does not widen its lead over PPRM.
Noisy-but-Valid (Feng et al. 2026, a calibrate-the-current-judge endpoint test) makes no false certification but also certifies none of the eight safe endpoint cases.

A non-LLM evaluator replacement on DICES (10k-feature unigram → 30k-feature uni+bigram TF-IDF) barely moves anything (portfolio 1,184.8 vs 1,182.2): a modest replacement left history informative.

What this changes for release practice#

Label each certificate by what history did. "History entered validity (transport assumed, radius ρ declared)" and "history only steered sampling" are different claims with different error accounting.
An LLM-judge rubric or version change is an M0 event until labels say otherwise. No amount of judge-score monitoring can detect a residual shift (Prop. 2); only purchased labels can. This is the formal ground under Li's empirical rule not to transport old-version calibration.
Budget vigilance as a floor, not an overhead. Theorem 1 prices the labels a stationary, nothing-changed release still has to buy.
Random sampling is a validity assumption, not a convenience. The paper's ethics note is explicit: audit rows chosen by convenience can produce a false certificate that ships a harmful update.

Evidence notes#

Tier empirical, kept. Full proofs, a protocol fixed before the final roles were analyzed, matched comparators sharing pools, orders and budgets, block-bootstrap intervals, and released code, frozen judge outputs and per-block results (per the reproducibility statement). Four authors from four universities; no vendor COI. The AI-use statement discloses generative-AI help with methodology critique, code, statistical interpretation and drafting, with authors checking every claim.
The LLM-judge evidence is the weakest part. Both LLM studies were designed after the primary protocol was fixed; they change one fixed model's rubric, not the model; the repeated-judge study ran 5 random orders where its written protocol planned 50 (orders are averaged within block, so the inferential unit is unchanged); one of six comparisons is a tie.
"PPRM-style" is the authors' adaptation of Zhang et al. (2026) to a fixed finite pool, not the original procedure, and its theorem is not transferred.
The lower bound is for i.i.d. Bernoulli draws, not finite pools without replacement; in a scaling study the portfolio uses 4.06–4.80× the bound (Table 7). Proposition 6 and Corollary 9 hold only in the canonical model. Error control is block-level.
"Zero observed false certifications" is an observation, not an error estimate; validity rests on Theorem 4's assumptions.

Connections#

Version-Dependent Judge Error — the empirical counterpart. Li finds that carrying an old agent version's judge calibration forward multiplies comparison error 5.2× and prescribes a paired audit of current outputs. This page proves why no label-free check could have rescued the transport, and shows the old calibration is still worth keeping as advice inside a valid certifier
LLM-Judge Validation — Yang et al.'s "replacing the instrument is a measurement-validity event" (cited by this paper as the reason stale history can be costly), now with a decision rule: a rubric change in one fixed judge moves scores beyond run-to-run variation, and the certifier pays for it in efficiency, not validity
LLM-as-a-Judge — the thresholded use (ship/no-ship) that page's open question asks about. Here the judge is admitted to a threshold decision only as a proxy whose residual is bought with labels
Recurring Production Agent Evaluation — the evolving-system release gate, with the label cost of re-certifying each release made explicit; a calibration ledger carried across releases is exactly the M2 advice this certifier hedges
Benchmark Score Redundancy — the same design rule from the estimation side: CollabEval feeds historical model runs in as control variates so they buy efficiency while validity survives the history being wrong. Portfolio vigilance makes the same separation for a sequential release decision
Certified Test-Time Scaling Curves — a sibling betting-confidence-sequence audit (anytime-valid, label-efficient, measured against matched certified baselines). Both papers report cost ratios against the cheapest valid competitor, not against an uncertified one
Adaptive Stopping in Evaluation Sampling — the Bayesian counterpart of sequential stopping. optstop buys interval width under the stopping-rule principle; this certifier buys frequentist anytime validity via Ville's inequality, at the cost of the vigilance labels Theorem 1 prices
LLM-Judge Self-Preference — the size of a judge-model swap on a fixed item set, which this page's second open question asks about. On the same Text Arena battles, one author's answers are rated +55pp over people by one judge and +3 by another (Astra's answers under Astra vs DeepSeek V4 Flash). A ledger of residuals from one judge model would not transport to the other

Open Questions#

Can transport itself be certified with few labels when the bridge is only partly testable? The paper names this as its first open problem. A cell-verifying certifier (VIGIL-CELL) held zero false certification on all 120 Folktables injected blocks but was costlier than PPRM in aggregate and was evaluated post hoc.
Does the portfolio's advantage over ledger-only survive a judge model change (not a rubric change), and an agent or model update graded by a frozen judge — the setting where Version-Dependent Judge Error finds differential error? Both LLM studies here hold the judge model fixed and change only the rubric.
Is the vigilance price for finite pools sampled without replacement closer to the i.i.d. bound than the 4–5× gap measured here? The paper names this as its second open problem.

Sources#

When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift — Joyanta Jyoti Mondal, Mridul Banik, Md. Shifatul Ahsan Apurba & Md Masud Al Mahmud (Univ. of Delaware / UAB / Indiana Univ. Indianapolis / BRAC Univ.), When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift, arXiv 2609.32267, 2026-09-26, 27pp, empirical. Cited for: Abstract; §1 (M0/M1/M2 framing); §3 (Eq. 2–3); §4 (Theorem 1, Prop. 2, Theorem 3); §5 (Theorems 4–5, Prop. 6); §6–7 (all headline results); §8 Limitations; Appendices E (matching certifier, label caps), F (assumption necessity, stale-advice numbers), H (per-block, external), I (corrupted advice, scaling), J (sensitivity, non-LLM evaluator change), K (LLM rubric studies, transport price, Noisy-but-Valid), L (Folktables). Figures 2 and 7 were viewed and match the prose. Parse warning: docling-derived (2.126.0 / docling-mlx 0.1.1, confidence_grade: excellent, 13 tables). The table-collapse flag is Table 12, whose DICES-990 rows are welded into one cell ("Ledger-only Robust-only PPRM-style WoR Hoeffding", differences split as - 109. / 1); it was recovered with pdftotext -layout (Ledger-only −140.7 [−256.2, −46.2]; Robust-only −109.1 [−199.3, −32.6]; PPRM-style CM-EB −239.1 [−394.4, −114.2]; WoR Hoeffding CS −503.3; all 9/9 pairings). Table 3 (notation) is collapsed into one row and not cited. Tables 1, 2, 5, 6, 7, 10 and 13 were checked against the prose and are clean; every number above is from prose or a reconciled row.
Self-Preference of AI Judges — Arena.ai, Self-Preference of AI Judges, blog post, 2026-09-30 (empirical; COI: Arena sells human-preference evaluation). 12 judges × 1,460 Text Arena battles, 34,580 verdicts. Full treatment on LLM-Judge Self-Preference; cited here for the judge-swap magnitude in the Connections bullet.