Shortcut Audits of Failure-Attribution Benchmarks

Sources#

• Auditing Failure Attribution: Label Predictability and Judge Behavior

Summary#

A failure-attribution benchmark asks a judge which step or agent caused a failed run (Automated Failure Attribution). A high score is meant to show the judge checked the faulty step against the rest of the trajectory. Xu & Gao (Auditing Failure Attribution: Label Predictability and Judge Behavior, Purdue, arXiv 2610.17747, 2026-10-13, empirical) audit three recent benchmarks (WHO&WHEN PRO, AFTraj, Aegis) and find that on every one, a predictor that never compares two steps matches or beats GPT-4.1. The cause differs per benchmark: a text cue left by error injection, a failure mix that decides the ranking, and a lenient metric.

The paper's contribution is a four-test audit that needs no new annotation:

The paper's Section 7 argues the tests are complementary: on each benchmark, one test or metric used alone would have given the wrong conclusion.

WHO&WHEN PRO: the injection leaves two cues#

One step read alone finds every ALFAgent error. A logistic regression on TF-IDF features of a single step, with field names removed, picks the annotated step in all 311 ALFAgent trajectories from the full text or the reasoning field alone. Without the reasoning field it finds 21.2%. Deleting recurring phrases such as "Repetitive loop" leaves it at 98.4%. Its largest positive weights go to words that declare progress or a loop, such as already, can, keeps, so and now. The cue is not specific to one system: on smolagents (3,276 trajectories), where reasoning runs to the final step, the same classifier finds 95.2% from reasoning alone and 90.8% of errors that are not at the final step. The authors cannot tell whether that text states the mistake plainly enough for a careful reader to spot it, or carries traces of how the error was injected. AFTraj's word-versus-structure experiment (below) points toward an injection artifact.

Where the reasoning stops is the second cue. WHO&WHEN PRO injects an error by adding an instruction to the agent's prompt at one step and then resuming execution. In ALFAgent the resumed steps carry no reasoning. So "pick the last step with reasoning" is right on 310 of 311 without reading a word. A position rule (always final or penultimate) gets only 2 of 311 on ALFAgent, but 51.4% across all 6,257 text trajectories and 78.8% on MathChat. Position is a strong cue on most of the benchmark's other text systems.

This is the construction Automated Failure Attribution describes as making labels trustworthy: a warm-started success, a single injected action generated by the agent's own model call, and a filter for traces that "leak construction artifacts". The labels are still correct by construction. The audit shows they are also predictable from a single step, and that the artifact filter missed both cues.

Zero-shot judges do not use the cues. GPT-4.1, Claude Sonnet 5 and o3 score 58.8 / 57.2 / 56.3% on ALFAgent under the authors' prompt. Three results show they are not riding the cues:

• Moving the boundary does not move them. Adding one generic sentence to every empty reasoning field after the error ("Template continued") moves 1 of their 536 correct answers to the new boundary. Filling those fields with task-related reasoning ("Task continued") moves none of GPT-4.1's or Claude Sonnet 5's 361 correct answers, and costs 4.2 and 9.0 points.
• Rewording the error step changes nothing detectable. GPT-4.1 moves +1.4 points and Claude Sonnet 5 +0.4, and every interval includes zero.
• They do worse when the error announces itself. On the 53 records whose error-step reasoning begins "Repetitive loop", GPT-4.1 is right on 8 (15%) against 175 of 257 (68%) on the rest. Claude Sonnet 5 gets 10 (19%) against 168 (65%).

They do need the reasoning: removing it costs 43.7 / 42.1 / 45.7 points. Residual accuracy credits them with nothing beyond the cues, because the cue predictors miss almost nothing. The honest reading of the zero-shot WHO&WHEN PRO numbers is therefore low but not cue-inflated. Under the benchmark's own prompt the step accuracy is lower still (GPT-4.1 50.8%, Claude Sonnet 5 51.4%), Joint is 25.1% and 22.8%, and the agent is named correctly in 100.0% and 97.4% of trajectories.

Judges trained on the labels learn the boundary. The authors LoRA fine-tuned Qwen2.5-1.5B and Qwen3-8B on ALFAgent's labels (task-disjoint folds, two seeds each). All four runs reach 99.7–100%, 41 points above GPT-4.1. Task continued then costs them 97.7 and 99.0 points (1.5B) and 35.7 and 30.9 (8B). The same text costs the un-tuned Qwen3-8B 4.5 points, and costs the boundary-blind classifier 9.0. Two controls rule out the added words themselves: appending the sentence before the error (keeping the boundary) costs nothing, and the classifier's top cue words are rarer in the added passages than at error steps.

The tuned models depend on the cue without following it. At most 8.4% of their answers move to the new boundary (step 30). Lost answers land past the error, mostly near the end; for the 1.5B model, 216 and 175 of them land on step 29. A follow-rate statistic alone would have cleared them, and only the accuracy drop exposes the dependence. That is the paper's sharpest methodological point: cue use need not look like cue following.

AFTraj: the tie is set by the failure mix#

On AFTraj's 163-failure first-alarm test (108 injected, 55 natural), current-turn TF-IDF and GPT-4.1 are each right on 82 (50.3%) but share only 36. GPT-4.1 gains 46 and loses 46, with residual accuracy 56.8% [45.8, 67.6]. Split by source, TF-IDF leads on injected failures by 17.6 points [4.0, 31.5] (65.7 vs 48.1) and trails on natural ones by 34.5 [15.4, 52.0] (20.0 vs 54.5). The two gaps cancel at the released 66.3% injected share. This is Aggregate Cancellation on a task-accuracy outcome: identical accuracy over largely disjoint correct sets, with the stratum weight as the control variable.

The top rank is equally fragile. Probe (a linear classifier on frozen Qwen3-8B hidden states) leads DeepSeek V4 Pro 62.0 vs 60.7, but only when injected failures exceed 63.0% [38.1, 92.0] of the mix. At equal weights DeepSeek leads by 4.9 points. Neither the released share nor the full corpus share (65.4%) lies outside that crossing interval, so the benchmark cannot rank its two best methods.

Source is confounded with domain (every coding failure is injected), and on the 68 agentic failures alone every injected-minus-natural interval includes zero. A separate within-domain run on 468 agentic failures gives a cleaner result. A word classifier beats a structure-only classifier (role, position, length, thought/action markers) on injected failures, 64.2% vs 44.6%, but not on natural ones, 58.1% vs 63.1%. So a turn's words carry signal beyond its structure only for injected errors. That is the AFTraj counterpart of the ALFAgent lexical cue, and the best evidence in the paper that the cue is an injection artifact rather than ordinary mistake legibility. On the 46 natural agentic failures, zero-shot Qwen3-8B beats both trained classifiers, by 21.7 points over Probe and 34.8 over TF-IDF.

For the injected-is-easiest reading. Automated Failure Attribution treats single-injected-error corpora as the easiest setting and so as an upper bound on organic debugging. On AFTraj that holds for the cue-reading classifiers (+45.7 and +46.9 points injected over natural) but not for the zero-shot judges. GPT-4.1 scores 48.1 injected vs 54.5 natural; Qwen3-8B 38.0 vs 58.2 and Qwen3-32B 26.9 vs 50.9, both intervals excluding zero. Injected failures are easier for shortcut learners, not for judges, though the domain confound bounds how far that transfers.

Aegis: a metric that pays recall for false alarms#

Labelling every speaker faulty, a predictor that sees only the speaker list, scores 49.2% micro-F1 against GPT-4.1's 50.0% (gap 0.8 [−1.3, +2.8]). The benchmark's own macro-F1 ranks the predictor above GPT-4.1 (18.1 vs 15.3). Exact-set accuracy separates them: 1.5% vs 20.7%, a 19.2-point gap [15.6, 22.8]. The predictor's precision is 37.8%, and each innocent agent flagged is a cost to a developer that micro-F1 does not charge. Two data-hygiene findings come with it. First, 417 annotated name occurrences match no listed speaker, and 275 of those differ only in case or whitespace; after normalisation GPT-4.1 leads by 4.3 points on micro-F1 and 19.5 on exact sets. Second, the public data has changed across revisions: 107 of the 600 audited conversations are gone from the current Hugging Face release, and the original paper does not name the revision behind its scores.

Video: position beats the judge#

On WHO&WHEN PRO's video subset, always answering the final step scores 63.3% (mean of DVD and EVA) against GPT-4.1's 59.8%. On EVA every label is the final (400) or penultimate (348) step, and GPT-4.1 answers the final step in 744 of 748 trajectories. It finds 2 of the 348 penultimate errors. On DVD, residual accuracy alone would overrate GPT-4.1 (it finds 27 of the 45 errors the guess misses, 60.0%), but it also misses 39 that the guess gets, and trails it by 7.1 points.

What to take from it#

• A score certifies cross-step checking only after the restricted-input baselines are reported next to it. The paper recommends benchmark developers publish its Table 5 (one row per benchmark, one column per test) beside the headline score. This is the attribution-benchmark version of the hypothesis-only NLI baselines (Gururangan 2018; Poliak 2018), with Feng et al.'s 2019 caution built in as the cue-following test: a working partial-input baseline shows a cue exists, not that a model uses it.
• Training on injected labels is where the risk becomes real. Zero-shot judges here ignored the cues. A judge trained on the labels found them within three epochs, at 1.5B parameters. Any "small trained attributor beats frontier judges" result is the case that needs a cue-following arm, which bears on Long-Horizon Agent Failure Signature's Scout. Scout is trained on organic labels with no injection boundary, and its transfer to a held-out domain and its best-of-N gains are evidence a pure cue learner would not produce. It has not been tested against position or wording cues.
• Injection buys correct labels, not cue-free ones. The warm-start construction makes the decisive step exact. It does not stop the injected instruction from leaving a lexical signature at that step, or the resumed run from leaving a structural one. These are two different properties, and the benchmark validated only the first.

Evidence and conflict of interest#

empirical, confirmed on full read. The authors are two Purdue researchers, independent of all three audited benchmarks. Intervals are 95% bootstraps over task groups (10,000 resamples). Restricted-input predictors are selected on held-out halves or task-disjoint folds, never on the records they are scored on. The total compute was about 20 GPU-hours. Self-reported limits: cue following was run only on ALFAgent; judges answer the authors' prompt, not the benchmark's (except one experiment); fine-tuning covers one subset, two sizes and two seeds; Aegis and video use GPT-4.1 only; and scoring uses released annotations only, so a judge blaming a different but valid step counts as wrong. Two judge-as-instrument overlaps, both disclosed: DeepSeek V4 Pro writes the intervention text and is also an AFTraj judge (on a different benchmark), and GPT-4.1's automatic paraphrase-equivalence check is not independent of GPT-4.1 as a judge. The 60 human-reviewed pairs (56 equivalent, 4 minor, 0 material) are independent.

Connections#

• Automated Failure Attribution — the benchmark family audited here, and the page whose "labels can be trusted" and "upper bound" readings this audit qualifies: correct labels, but predictable from one step, and judge accuracy not higher on injected failures on AFTraj
• Benchmark Task Defects (Spec–Test Mismatch) — a sibling defect class. There, the task's prompt and tests disagree, so a pass stops meaning success. Here, the task is well-posed but its label is recoverable from surface features, so a high score stops meaning the capability was exercised. Both are found by auditing the benchmark rather than the model
• Aggregate Cancellation — the AFTraj tie is a measured task-accuracy instance: 82/163 each, 36 shared, +17.6 injected and −34.5 natural, cancelling at a 66.3% mix, with the crossing weight as the identified control variable
• Long-Horizon Agent Failure Signature — Traverse is an organic-failure benchmark (no injection boundary to learn), and Scout is the trained small verifier this page's fine-tuning result says should be cue-tested before its lead over frontier judges is read as capability
• LLM-Judge Validation — residual accuracy and cue following are judge-validation steps that agreement statistics do not cover. A judge can agree with the labels at 99.7% because it learned how they were made
• Benchmark Convergent and Discriminant Validity — the same lesson about method choices across benchmarks: there, sharing an LLM-judge scoring method predicts agreement more than the shared concept does; here, micro-F1 versus exact-set and the injected/natural mix decide the ranking within one benchmark
• Verification as the New Bottleneck — the measurement side of the bottleneck: an attribution leaderboard can be topped by a model that never verifies anything
• Benchmark Integrity Maintenance (Unearned Passes and Channel Closure) — the same audit-the-benchmark stance on agentic coding and tool-use suites, by reading passing trajectories rather than restricting inputs, and carried through to a patch whose acceptance requires fresh runs

Open Questions#

• Does a trained attribution verifier on organic-failure labels survive a cue-following test? Scout (Long-Horizon Agent Failure Signature) beats six frontier judges on Traverse. Organic labels have no reasoning boundary, but position and progress-claim wording may still predict the first mistake. Falsifiable: run this paper's input-restriction predictors on Traverse, then move the strongest cue with the mistake fixed and measure Scout's accuracy change against an un-tuned base.
• How much of WHO&WHEN PRO's headline text step accuracy survives residual accuracy against the per-system position rule? The position rule alone is right on 51.4% of all 6,257 text trajectories and 78.8% on MathChat, against a best-judge 73.9% on the paper's own text set. Per-system judge outputs scored on the records the position rule misses would show how much of the leaderboard is above the position floor.
• Can the injection pipeline be fixed so labels stay exact but stop being predictable? Candidates: regenerate the error step's reasoning without the injected instruction in context, and keep reasoning on the resumed steps. Falsifiable: rebuild ALFAgent and smolagents and check that the independent-step classifier and the boundary rule fall to their position-rule floor while the decisive-step guarantee holds.

Sources#

• Auditing Failure Attribution: Label Predictability and Judge Behavior — Tianyang Xu & Jing Gao (Purdue University), Auditing Failure Attribution: Label Predictability and Judge Behavior, arXiv 2610.17747, 2026-10-13, empirical, 26pp docling parse (25 tables, 4 pictures). Sections used: Abstract, §1, §3 (the four tests and Equations 1–3), §4.1–4.4 (ALFAgent and smolagents input restriction, cue following, paraphrase test, fine-tuned judges), §5 (AFTraj), §6 (Aegis), §7, Limitations, and App. A, B.1–B.9, C.1–C.4, D.1–D.3, E. Figures quoted from prose and captions. Table 10 (ALFAgent interventions) and Table 13 (fine-tuned judges) were cross-checked against the prose figures they support.
• Parse warnings. (1) Table 1 shifted: "step position" is the only restricted input for the All text row, and "presence of reasoning; step text; reasoning length" belongs to ALFAgent. The parse slides the ALFAgent inputs into the row above. Verified with pdftotext -f 2 -layout; not cited row-wise. (2) Table 15A's two "Between methods" rows are welded into one grid row (Probe - DeepSeek TF-IDF - GPT-4.1). The PDF (pdftotext -f 23 -layout) gives Probe − DeepSeek = +1.2 / +13.9 / −23.6 and TF-IDF − GPT-4.1 = 0.0 / +17.6 / −34.5 (released / injected / natural). The values quoted here match the §5.2 prose. Table 15B's between-methods values are dropped from the parse (Qwen3-8B − Probe +21.7 [4.7, 38.6], Qwen3-8B − TF-IDF +34.8 [14.3, 54.5]) and are quoted from the App. C.3 prose. (3) The Table 5 cell "Independent-step classifier 311/311 (95.2% on smolagents)" flagged by the ingest checker is not a collapse: the PDF cell holds both values. (4) The App. C.3 domain table is parsed from Figure 4 with its header row at the bottom; its values match the caption and prose (Probe − TF-IDF coding gap +29.7) and are not cited beyond that.