Automated Failure Attribution

Sources#

• An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures
• TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
• Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?

Summary#

Automated failure attribution is the task of reading an agent trajectory that ended badly and answering three questions: who (which agent produced the decisive action), when (which step), and why (which failure mode). It is the assumed-available capability underneath every "we'll debug it from the traces" story in this vault, and the load-bearing feedback signal for self-evolving agent loops — the whole point being that a failure becomes actionable only once you know what to change.

Liu, Xi, Zhang et al. (Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?, Penn State / AG2ai / MBZUAI / Oregon State / Mathos AI, arXiv 2607.09996, 2026-07-10, empirical) build the largest corpus of the task — 12,326 failed trajectories, 26 benchmarks, 15 agent frameworks, 9 task categories, 18 error modes, 3 modalities (text, image, video), traces averaging 7.5 steps and 1,139 words with a 50-step maximum — and title the paper with the question can LLMs really attribute failures in AI agents?

The measured answer is mostly no, and the shape of the "no" matters more than the headline. Ten frontier models, best-in-column on text:

Metric	What it asks	Best (text)	Range across 10 models
Agent	which agent is responsible (MAS traces only)	57.5 (Qwen3.5-122B)	48.4 – 57.5
Step	exact-match index of the decisive step	73.9 (Qwen3.5-122B)	63.9 – 73.9
Error	macro-F1 over the 18 failure modes	22.2 (GLM-5)	10.8 – 22.2
Joint	all three correct on the same trace	25.3 (GLM-5)	16.2 – 25.3

For reference, the human panel ratifies the pipeline's own labels at 94.0 / 90.0 / 90.0 (step / agent / error) with Fleiss κ = 0.73 and only 2.0% of traces judged to have no clear decisive error. So the labels are not the problem.

No single model dominates: Qwen3.5-122B leads agent and step, GPT-5.4 is strongest on step among closed models (72.3), GLM-5 takes error-mode F1 and joint accuracy. The paper's reading is that attribution "exercises a fundamentally different capability from general knowledge answering or instruction following" — which the flat cross-model spread supports: the gap between best and worst on text step accuracy is 10 points across a closed/open-weight frontier lineup that differs by far more than that on ordinary reasoning benchmarks.

Why the labels can be trusted: injection into a warm-started success#

The method is the paper's other contribution and it is the reason the numbers above are readable as a capability measurement rather than a label-quality artifact.

Prior injection-based benchmarks re-run the task from scratch and inject an error during the new rollout. Because LLM generation is non-deterministic, the new rollout drifts from the original before the injection point, so the decisive-step label is only approximate. WHO&WHEN PRO instead warm-starts: take a trajectory that succeeded, replay steps (a₁ … a_{t−1}) through the real framework pipeline (tool parsing, code execution, environment interaction) to rebuild both context and environment state, substitute a single corrupted action ã_t, and let the original system run from t+1 onward unaided.

Since the injected error is the only controlled change that turns a successful trajectory into a failed one, the resulting task failure can be attributed exactly to the injected agent and step.

Reverting ã_t recovers the successful seed, which makes t the decisive step by definition under Zhang et al.'s decisive-error formulation (the earliest index whose correction flips failure to success). Supporting machinery worth copying: static tool calls (web search, page retrieval) are served from a content-addressable cache populated during seed collection so observations are byte-identical on replay; stateful environments like live browser sessions get action-level fidelity checks that abort the attempt on divergence. The injected action is constructed in two stages — a frontier model writes an adaptive injection prompt from the seed context, which is then patched into the agent's own model call so the base agent generates the error in its own style. Traces that leak construction artifacts, or where the correct answer was already salient before step t, are filtered out.

The taxonomy is human-authored, not synthetic: Ph.D.-level reviewers inspect naturally failed traces per (agent, benchmark) pair, and their failure-mode profiles constrain which errors the generator is allowed to inject where. Humans define the label space; automation only populates it.

The consequence for reading everything below. Every trace is a single injected error into a run that was otherwise known-good, with a decisive step guaranteed to exist and to be unique. That is the easiest possible attribution setting — organic failures have multiple interacting causes, no guarantee of a single flip-the-outcome step, and no known-good remainder. Qualified 2026-08-13 on the last clause: where a failure axis leaves one arm of a matched pair succeeding, an organic failure does come with a known-good remainder for free, and the multilingual planning source below is built on exactly that filter. The measured accuracies are therefore best read as an upper bound on real post-mortem debugging, not a floor.

The three difficulty axes#

Trace length dominates. Step accuracy falls from 94% on traces under 3K tokens to 50% on traces exceeding 12K tokens, consistent across all ten models. The sharpest decline is between the 3–6K and 6–12K buckets — "roughly where traces transition from single tool calls to multi-step sequences with interleaved observations." The stated mechanism is the useful one: as more correct steps accumulate around the error, isolating the decisive one becomes harder. This is length degradation on a task where nothing is being retrieved and everything is in the window (Context Window Smart Zone).

Modality splits localization from diagnosis, in opposite directions. Step accuracy is highest on text (~69% averaged) and drops on image and video; error-mode classification runs the other way, rising from ~17% on text to ~37% on video, because "visual and behavioral cues provide diagnostic signal that pure text traces lack." A text-only benchmark — which every prior failure-attribution benchmark was — masks that difficulty profile entirely. One unexplained pattern to note rather than build on: the Agent column is 20–30pp higher on video (70.1–85.1) than on text (48.4–57.5), and the paper offers no mechanism for it.

Localization is coarse but not random. Under relaxed tolerance, the Step@0 → Step@1 gain is large (+13.9pp text, +16.9pp image, +28.1pp video) while Step@1 → Step@2 is small (+4.4 / +6.0 / +2.6). Models "either identify the correct neighborhood or miss entirely." So ~86–88% of text traces are localized to within one step even though only ~74% are exact — usable as triage narrowing, not as ground truth.

The finding that actually threatens multi-agent debugging#

Error-mode confusion is not uniform. Perception, reasoning and action errors are recognized reliably — they leave clear signatures in observations or tool calls. Planning, verification and coordination errors are systematically relabeled as reasoning errors, to the point that on multimodal traces more than half of non-reasoning errors are absorbed into the reasoning label. The same off-diagonal mass shows up in the human correction matrix, weakly, in the same direction.

The mechanism is stated cleanly and generalizes past this benchmark:

These failures originate early in the trace but propagate through later steps, producing symptoms that resemble reasoning mistakes by the time they become visible. … current models classify based on the most salient symptom rather than tracing the causal chain back to the decisive step.

An ineffective plan produces a sequence of individually plausible but misguided steps. A skipped verification lets an early mistake compound. In both cases the local evidence at the point of failure looks like bad reasoning, and the root cause lies further back.

The implication for anything built out of coordinated agents: the failure families that exist only because you built a multi-agent system — delegation and orchestration errors, communication failures, over-reliance on another agent's revision — are exactly the ones an automated attributor cannot name. It will hand back "reasoning error" and point at a downstream step. A team acting on that post-mortem is steered toward "use a stronger model" when the fix is orchestration. The 18-mode taxonomy is six families deep (perception 2, reasoning 4, planning 2, action 5, verification 2, coordination 3) precisely so this distinction is expressible — and the classifier collapses it anyway.

Read the whole trace at once#

Three protocols, compared on a stratified 1,444-trace subset:

• All-at-once — full trajectory in one prompt, predict agent/step/mode jointly.
• Step-by-step — read sequentially, stop at the first step believed decisive.
• Binary search — recursively bisect the trajectory to localize.

All-at-once wins on step accuracy by 7.6pp over step-by-step (63.2 vs 55.6 averaged across the ten models) and takes the highest Joint accuracy for eight of ten models. Step-by-step's specific defect is early commitment: its prediction-distance distribution is systematically shifted toward steps before the true error, because a sequential reader "locks onto the first suspicious step and cannot revise after observing later context." Binary search is directionally unbiased but equally imprecise — slicing removes the context needed to judge the slice.

This is the same shape as the confusion result. Attribution is a global judgment about a causal chain; every protocol that makes it locally, on a prefix or a fragment, degrades. And it is also the cheap answer: all-at-once dominates the Pareto frontier in all three modalities, with Qwen3.5-122B reaching the highest text accuracy at seven times cheaper than GPT-5.4 and Gemma 4 leading on video. There is no cost argument for the incremental protocols.

Giving the judge the answer makes the diagnosis worse#

The counter-intuitive result, and the most transferable one. Supplying the task's ground-truth answer (w/ G) to the attribution judge — mirroring a verifier-assisted loop where outcome feedback is available — improves perception-error attribution and degrades or flattens reasoning-error attribution, across all task domains and frameworks.

The paper's explanation is an outcome-vs-process split. Perception errors (bad OCR, misread chart) are hard to check without a reference, so G fills a genuine gap. Reasoning errors leave a traceable logical chain, and G "tempts the judge to shortcut via answer comparison rather than tracing process." Two paired case studies show the mechanism directly:

• A smolagents run retrieves the correct Pokémon TCG Pocket release date, then reinterprets "arrive" as anniversary content. Without G, GPT-5.4 and Claude Sonnet 4.6 both trace the process and correctly call it task misunderstanding. With G = 2024-10-30, both notice the agent "had already retrieved the correct release date," reclassify as reasoning error, and miss that the agent answered a different question.
• A 4-step image trace where the agent fabricates "Ray-Ban Meta smart glasses" against visible GENTLE MONSTER branding, then self-corrects to the wrong collection. Without G, Claude Sonnet 4.6 and Gemini 3 Flash both flag the hallucination at Step 2 as the decisive error. With G, both skip Step 2 entirely and diagnose the downstream collection confusion as a reasoning error. "The gold date made it easy to verify which date is wrong but drew attention away from where the reasoning first broke."

More outcome signal makes the judge a better outcome verifier and a worse process verifier. That is a direct caution for any self-improvement loop that feeds the gold answer to its own post-mortem step in the belief that more information is strictly better.

A quiet architectural precondition#

Buried in the implementation appendix: the base agent models are GPT-4.1 and Gemini 3 Flash, and GPT-4.1 was chosen because "its reasoning chains are interpretable and unencrypted, which is essential for step-level failure attribution since the attribution judge must read intermediate reasoning to localize the decisive error."

Attribution is a capability with a hard dependency on trace legibility. Every step toward summarized, encrypted, or discarded reasoning traces in frontier deployments removes exactly the evidence this task consumes — and this benchmark's numbers were obtained on the legible case.

A non-judge instrument for the "when" half — and the input it needs#

Every number above is a judge reading a trace. There is another way to localize the decisive step, and it comes from the RL side rather than the evaluation side. TRACE (Tao et al., UW–Madison + Microsoft Research, arXiv 2607.13988, 2026-07-15, empirical) scores each trajectory prefix by how predictable a frozen reference model finds the gold answer, converts that into a "fraction of the initial gap closed" value, and reads the per-turn credit off adjacent differences. It was built to supply dense reward, but the same values are a measurement: define an answer-secured prefix as V ≥ −0.3 and the decisive turn falls out numerically. In its five successful traces one tool call closes most of the gap (credit sequences like [+6.39, +0.59, +0.10, +0.05]); in its five failed ones the trajectory reaches an answer-secured prefix and then loses it after one diagnostic tool call. No judge, no reasoning to interpret, no 18-way taxonomy — a scalar per turn.

The precondition is exactly what this page's task lacks. The probe is defined by conditioning on the gold answer, available at training time and absent when you are debugging a failure in the wild. So this is not a cheaper judge; it is a stronger input to an easier version of the problem, and the two occupy different slots: gold-answer-conditioned credit for the training loop, post-hoc attribution for everything downstream of it. The comparison is still worth holding for two reasons. It is another instance of this page's recurring shape — where something can be counted, count it, and fall back on a judge only where nothing can — alongside the countable stage boundaries of FCAR = MER × ECAR and the deterministic L0–L3 taps. And it suggests a variant of the legibility question below that runs the other way: a log-probability probe localizes the decisive turn from what the observations did to answer predictability, never reading the agent's reasoning at all, which is precisely the evidence frontier deployments are removing. Nobody has run that as an attribution method on unlabelled traces, and without a gold answer it is unclear what would replace the conditioning target.

A collected failure set instead of an injected one — and the convention that keeps the question open#

Every number above rests on failures that were put there. Pahuja, Brokman, Hofman et al. (An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures, Fujitsu Research of Europe / Cohere / Fujitsu Research, arXiv 2608.03735, 2026-08-04, empirical) is the corpus's first source whose failure set is collected rather than synthesized, and it earns its place here on provenance rather than subject — the subject is multilingual planning; the methodological interest is that nothing was injected. §3.2 states the sampling rule in one line:

Plan extraction: We filter cases where the development agent succeeds on an English query but fails at a non-English query.

Real agents (Qwen2.5-32B-Instruct and Cohere Aya-Expanse-32B) inside a real framework (HuggingFace's Open Deep Research) on real GAIA-derived tasks. 80 failed pairs were read one at a time against the successful English plan and the original question, stopped at saturation — when new samples yielded only previously observed categories — by one primary analyst with three researchers confirming each assignment. The derivation set is deliberately disjoint from the evaluation in both models and framework, so none of the three backbones and none of the harness that later shows the gains contributed a single taxonomy sample.

The sampling rule is worth separating from the paper's own framing, because it recovers for free the property this page's benchmark had to buy with injection. WHO&WHEN PRO needs a warm-started success to make the decisive step golden by construction; this filter gets a paired known-good run from the task itself — the English execution, same agent, same tools, same question. The "intervention" is a language swap, a variable real users vary rather than a synthetic corruption, and the diagnosis is the difference between the two plans. That is a natural experiment with the shape of an injection benchmark and none of the construction, and the trick generalizes to any axis where one arm of a matched pair succeeds.

The multi-fault half is not answered — it is defined away#

The provenance question above has two halves, and this source settles only the first. Appendix G, on the LLM judge that scales the categorization past the 80 hand-read cases:

When more than one failure occurs within a single agent plan, we instruct the judge to mark a single primary category, defined as the failure most responsible for the erroneous plan. This yields a mutually exclusive labeling scheme consistent with the convention used in our manual analysis, and avoids double-counting when reporting per-category and per-language distributions.

Read the last clause: the forced single label governed the original manual derivation too, not merely the automated scaling step. So the paper concedes that multi-fault traces exist and then assumes a primary cause and labels to it. It therefore neither claims nor demonstrates that an organic trace has one decisive fault — and the human validation validates the wrong thing for this purpose, checking judge-versus-annotator agreement on the assigned category (κ = 0.860, macro-F1 = 0.906) and never whether the "primary" pick was the sole cause.

That is the honest state of the question, and the convention is itself the finding: two independent research programmes now sit on either side of "does a decisive step exist," and neither measures it — one guarantees it by construction, the other instructs its annotators to produce it. The field routes around the problem rather than pricing it. The cheap experiment nobody has run is a multi-label arm: re-judge the same 2,689 already-labelled baseline failures (857 + 917 + 915 across the three backbones) with the mutual-exclusivity instruction removed, and compare per-category shares against the forced-single-label distribution. Nothing else in the setup has to change, and the fraction of plans carrying two or more grounding failures falls straight out.

The taxonomy that was converted into a mitigation#

What this source has that this page's 18-mode taxonomy does not is a demonstrated payoff. Actionability was a construction constraint, not a claim made afterwards — "for actionability we require a category to be specific enough to directly inform improvements to the plan" — so the five categories (entity, source, temporal, operation, answer-format grounding) each name a semantic commitment a plan must preserve, and each maps to a field. TART is that mapping: an LLM converts the query (in any language, without translating or answering it) into a JSON object with entities, time_constraint, source_constraint, attachment_type, operations and answer_type over closed vocabularies, which is then injected into the system prompts of the planner and the coordinator and worker agents rather than consumed once at plan time.

On GAIA-MAPS with the OWL workforce and the base model frozen, GPT-5-mini's mean EM goes 0.249 → 0.305 (+5.6 points) across eleven languages, improving ten of them (Russian is the exception at −0.006): German +9.1, Hindi +9.7, Arabic +7.9, Yoruba +10.9, Igbo +7.9. A second independent run means +4.7, with TART ahead in 20 of 22 language-run comparisons. Mistral-Large-3 gains +5.9 on the seven-language subset (every language up) and Qwen3-VL-235B-A22B +3.6 (six of seven up, one unchanged, none down); on a second dataset, MULTITAT with a three-agent system, Mistral gains +10.0 and Qwen +3.0.

The transferable part is not the size of the gain but that the loop closed at all. Both taxonomies here were built by reading real traces and both are six-ish categories deep; the difference is that this one was required to be specific enough to inform a fix, and the fix was then built and measured. That makes "actionable" a property you can falsify rather than an adjective in an abstract — and it is the standard the 18-mode taxonomy has never been held to, since its modes are scored for classifiability and used to repair nothing.

Where the gain comes from, and the part that may be exact-match compliance#

The ablation is the load-bearing table and it cuts against the headline. Cumulatively on five languages: 0.233 with the surface constraints only (entities, time, source) → 0.256 (+2.3) once the operations field is added → 0.287 (+3.1) with the full representation, +5.4 total. The larger of the two increments is the final one, and it adds only attachment_type and answer_type; the paper does not decompose it further.

That matters because answer-format grounding is defined as a case where the plan is substantively correct and only the output shape is wrong ("expected 116 rendered as 116000"), and it accounts for 159 of GPT-5-mini's 857 baseline failures — 18.6%, its second-largest taxonomy-covered category behind operation grounding's 206. Exact match scores every one of them as zero. So an unknown and possibly large share of a taxonomy-guided gain measured under EM is the representation telling the answer agent what shape to emit, not the planner grounding the request better. The direction and the cross-model consistency survive this; the interpretation of +5.6 as recovered planning capability does not, and no arm in the paper separates them.

The plateau: a repaired plan does not survive a long chain#

Averaged over all three backbones, TART gains +9.0 at Level 1, +5.0 at Level 2 and 0 at Level 3 — and the paper runs the right diagnostic instead of leaving it as noise. The obvious explanation would be fewer correctable planning failures at Level 3; the judge's own counts refuse it, with the taxonomy-covered share of failures flat at 55.2% for both Level 2 and Level 3 (60.4% at Level 1). What does change is intrinsic task length, from GAIA's English ground-truth annotations: reference solution steps rise modestly 5.48 → 7.48 and then nearly double to 13.00, with required tools 1.58 → 2.55 → 3.38. The paper's reading is the one to keep — even when the initial grounding failure is fixed and the constraints stay in view for the coordinator and workers, "a longer downstream execution chain still gives more independent opportunities for retrieval, tool-use, or reasoning errors to occur."

Set that against this page's strongest measured axis: attribution accuracy falls with trace length (94% under 3K tokens → 50% past 12K). The same variable degrades both ends of the loop. Diagnosis gets worse as the trace gets longer, and the payoff from acting on the diagnosis also gets worse — so the regime where you most need a post-mortem is the regime where the fix it recommends buys least. Neither paper connects the two; the pairing is the useful output.

The gradient, and the two things that could produce it#

The paper's headline diagnostic claim is a resource gradient: as Common Crawl share falls across the seven judged languages (English 40.58% down to Igbo 0.0012%; the full eleven-language set bottoms out at Nyanja's 0.0008%, five orders of magnitude below English), taxonomy-covered failures take an increasing share of failures while the residual Other category shrinks, with operation grounding becoming the largest single category for every model (24.0% / 36.5% / 30.2% of failures aggregated across languages for GPT-5-mini / Mistral / Qwen). The direction is strong, consistent across three model families, and visible in the per-language failure-share pies. Two confounds bound it, and the paper concedes both only in part.

The judge's validation is stratified on the wrong axis. The claim varies by language; the human study reports agreement by category — Entity 94.4%, Source 100%, Temporal 100%, Operation 83.3%, Answer format 85.7%, Other 88.2%, overall 88.9% on 117 decisive verifications. No per-language agreement figure appears anywhere, and the paper's own Limitations grants that "judge calibration may differ across languages, models, and failure categories." Worse, the two lowest-agreement categories are Operation (83.3%) and Answer format (85.7%) — precisely the two whose shares move most across the gradient, in opposite directions. A per-language calibration drift would produce a gradient of this shape with no change in agent behaviour at all, and there is nothing in the paper that separates the two. Judge reliability is known to degrade in exactly these languages (Reference-Free Judge Over-Crediting), which makes the missing stratification the specific check this result needed.

Six of the eleven languages are machine-translated, and they are the same six the taxonomy was derived from. Bengali, Swahili, Kyrgyz, Igbo, Yoruba and Nyanja were added by the authors via Google Cloud Translation; the taxonomy's 80 samples span Igbo, Yoruba, Bengali, Swahili, Kyrgyz and Nyanja — the identical set. So the failures are organic while the inputs producing them are not naturally authored, and entity grounding, the category that grows most in those languages (1–2 failures in English against 22–32 in Yoruba and Igbo), is the one an MT artifact would be indistinguishable from. Baseline and TART receive identical translations, so nothing biases the within-paper comparison; what it bounds is the diagnostic claim. One partial mitigation the paper never claims for itself: the judge's other rule explicitly captures "target language query intent, entities, sources are different from english query," which is where a mistranslated entity should land — but other is the category that collapses in exactly those languages, so either the rule rarely fires or MT damage is rare, and nothing here tells you which.

Connections#

• Zero Trust for AI Agents — these numbers price a security agenda item, not just a reliability one. The five-boundary isolation map's agent–agent defenses rest on attribution — "not just whether a multi-agent system failed, but which agent, which message, and which step," because containment is difficult without diagnosis. This page is the measurement that survey does not have: 48–58% responsible-agent identification, 16–25% joint, and the coordination and verification errors that exist only in multi-agent systems are the ones systematically absorbed into "reasoning error." Attribution-gated containment is being proposed on top of a capability that currently misses the multi-agent-specific failures most often (hub)
• Failures That Look Like Success — the detection question this page answers the next step of. That page argues the fix for silent failure is trace-level grading rather than output skims; this measures what a competent LLM extracts from that trace once you hand it over, under the most favorable conditions available — one injected error, an otherwise known-good run, a guaranteed unique decisive step — and gets the full triple right 16–25% of the time. Trace-level review stays the right prescription; automating it is not yet a solved subproblem
• Layerwise Omission Attribution — the same word, the orthogonal axis, and they compose. Rajan attributes a failure to where in the pipeline a fact died (nine layers, four of them deterministic software countable by canary diff with no judge); this attributes it to which agent and which step in a behavioral trajectory. Both find the honest split in the same place: what can be checked mechanically is checkable exactly, and what requires reading intent is where accuracy collapses. Rajan's L0–L3 taps need no model at all; this paper's task is entirely L4–L8 in that taxonomy, and 22% error-mode F1 is roughly the price of having no checkpoint to diff against. The pairing also supplies the missing half of each — a canary tap says the fact died at L1 but not which agent's decision caused the tool to be called that way; a decisive-step label says which action broke it but not whether the payload ever arrived. Extended 2026-08-13: the multilingual-planning source below adds a locus that page's nine layers do not have — the request-to-plan boundary, where a commitment dies with every byte intact — and its typed task representation is the canary protocol run prophylactically, with the free judge-less measurement (diff the plan against the extracted fields) never taken
• Stopping Under a Noisy Verifier — the same trust problem one level up, with a coefficient. Wu et al. price a noisy acceptance signal (Ā = ρ₀ + J·Q, so a low-J verifier's pass rate is mostly its own false-accept rate); this measures a noisy diagnostic signal, where the analogous defect is not a wrong bit but a plausible, confidently-argued wrong cause. Two direct couplings: the w/ G result is the process-vs-outcome verification split that page's four-parameter model assumes away, measured; and both papers land on the same counter-intuitive shape where more of the obviously-good input (more repair rounds there, more ground-truth signal here) makes the outcome worse
• Parallel Agent Orchestration — the debugging story this puts a number on. Agent-level identification is 48.4–57.5% on text multi-agent traces and step accuracy halves past 12K tokens, which is the regime a 64-agent campaign or a 5-concurrent-agent workflow lives in; and the coordination error family is the one systematically relabeled as reasoning
• Multi-Agent Collective Intelligence — the diagnostic tax on collectives. If a collective's advantage has to be established empirically, the instrument for asking why a run failed misattributes exactly the coordination-specific failures that distinguish a collective from a single agent, which biases any failure-driven iteration toward model choice and away from organization
• LLM-as-a-Judge — attribution is the judge paradigm pointed at a trajectory instead of an output, and it is the hardest instance in the vault: not "is this good?" but "which of 50 steps, which of 15 agents, which of 18 modes." The w/ G result is the sharpest transferable caution — adding a reference answer to a judge's prompt is normally a free improvement, and here it degrades the process-tracing half of the job
• LLM-Judge Validation — the validation practices this benchmark actually follows, unusually. Three independent annotators, majority vote, a chance-corrected Fleiss κ = 0.73 rather than raw agreement, an explicit "no clear decisive error" escape hatch (2.0%), and a published transfer matrix showing which families annotators moved labels between. That is close to the MVVP's prescription applied to label quality rather than judge quality; the gap is that the human panel ratifies pipeline labels while models predict from scratch, so the 94-vs-74 comparison is not a like-for-like human ceiling. Extended 2026-08-13: the multilingual-planning source repeats the ratify-don't-predict design (annotators are shown the judge's inputs and its verdict and asked to verify the assigned category), which makes two instances a pattern rather than a quirk, and it adds the defect the MVVP has no step for — a validation stratified by category underwriting a claim that varies by language
• Deterministic Pre-Execution Gates — the same failure class attacked before rather than after, and the comparison is stark. A read-only predicate over a proposed tool call that fires names the violated precondition, the step, and the agent with certainty and no inference; post-hoc LLM attribution over the same class of failure returns all three correctly on roughly one trace in four. Both papers reach the same conclusion from opposite ends — enforcement at the boundary beats diagnosis after the fact, and the reason to invest in gates is partly that the post-mortem is this weak
• Turn-Level Credit Assignment — the same which-turn-mattered question answered by a scalar instead of a judge: a frozen model's gold-answer log-probability differenced across each tool boundary, with the decisive turn falling out numerically. Not a substitute — it needs the gold answer, which post-hoc attribution never has
• Reference-Free Judge Over-Crediting — the reason the multilingual gradient above needs a per-language agreement figure it does not have. That page measures a judge looking well-calibrated in a high-resource language and unfit for the same task in a low-resource one (C1/C2 calibration gap 79–96 in English and Arabic, collapsing to 33 in Telugu, where one judge accepts ~60% of known-wrong answers), and reference-sensitivity rising monotonically with scarcity (NR→RC flip rates 0.29–0.37 English, 0.36–0.60 Arabic, 0.69–0.85 Telugu). Two transfers. The gradient result rests on a Claude Opus 4.8 judge run across languages down to Nyanja at 0.0008% of Common Crawl, validated by category and never by language — the one stratification that would separate an agent effect from a judge effect. And it runs the other way as a scope extension: that page's collapse is measured on correctness judging with a gold answer available, while this judge does category assignment with the English reference query and ground-truth answer both in the prompt, so whether low-resource degradation transfers from grading to classification is untested in either direction
• Verification as the New Bottleneck — the bottleneck extended past accept/reject. Deciding a run failed is the cheap half; deciding what to change is a separate capability with its own measured ceiling, and the vault's usual answer ("read the trace") is exactly the operation being scored here
• Production-Sourced Evaluation — the methodological contrast. This corpus is synthesized by injection into successful runs, which is what buys golden labels at 12,326 scale; the price is that its failure distribution is designed rather than observed, so it cannot speak to what actually goes wrong in production or how often
• Deep Research Agents — the same injection-for-golden-labels method used to get attribution for free, and the counterexample worth knowing about. MisKnow-Agent (arXiv 2607.20891, empirical) injects one misleading document into a deep-research run and decomposes the resulting failure as FCAR = MER × ECAR — was the document retrieved, and did an exposed report adopt it — an exact identity because no unexposed task ever adopts. That is stage-level attribution from two counters, no judge involved, and it separates a single 12-point framework gap into three different causes across three backbones. The lesson pairs with this page's: where a pipeline has countable stage boundaries, count them; post-hoc LLM attribution is the fallback for trajectories that don't. The two also find the same capability-without-invocation shape from opposite sides — there, the same models that adopt the misleading documents in-workflow unanimously flag them as misleading when handed one standalone; here, handing the judge the gold answer makes its process diagnosis worse. In both cases the model has what it needs and the harness fails to put it in the right place
• Agent Epistemic Vigilance — a controlled measurement of one of the coordination modes this page finds absorbed into "reasoning error." In hidden-profile tasks where the shared evidence favors the wrong option, groups of four vote the hidden-best choice 17–36% of the time against a solo ceiling near 100% — so "an agent abandoning its own correct answer after seeing another's" is reproducible rather than a rare trace artifact, and locally it looks exactly like ordinary reasoning, which is why the relabeling happens

Open Questions#

• Every trace is one injected error into a run that otherwise succeeded, so a unique decisive step is guaranteed to exist. Does attribution accuracy survive organic failures with multiple interacting causes and no single flip-the-outcome step — where the honest answer is often "three things were marginal and the fourth tipped it"? The human panel found only 2.0% of traces had no clear decisive error, but that is a property of the construction, not of agent failures. Until this is measured the 73.9 / 25.3 numbers are an upper bound of unknown tightness. Partially answered 2026-08-13 — the collection half only, by An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures (arXiv 2608.03735, empirical). The organic-provenance half is now sourced and the answer is that it is workable: 80 failure traces filtered by "the development agent succeeds on an English query but fails at a non-English query," real agents in a real framework on real GAIA-derived tasks, no injected faults, and a paired known-good run obtained from the language swap rather than from a warm start. The no-single-decisive-step half is untouched, and the way it is untouched is the finding: Appendix G instructs the judge that "when more than one failure occurs within a single agent plan… mark a single primary category," described as "consistent with the convention used in our manual analysis," so the multi-fault case is conceded to exist and then labelled away in both the automated and the hand-read arm. Human validation covers agreement on the assigned category (κ = 0.860) and not whether the primary pick was the sole cause. So the question narrows and gets cheaper rather than closing: re-judge those 2,689 already-labelled baseline failures with the mutual-exclusivity instruction removed and compare per-category shares — a multi-label arm over an existing corpus, no new traces needed, and the fraction of multi-fault plans falls out of it directly.
• Humans and models are never scored on the same task. Annotators ratify or correct a supplied label (94.0 / 90.0 / 90.0, κ = 0.73); models predict from scratch (73.9 / 57.5 / 22.2). So the ~20pp step gap is not a measured human ceiling, and could be much smaller or much larger. What is human from-scratch attribution accuracy on these traces? Cheaply falsifiable with the existing corpus and a blind annotation arm.
• The base agents were chosen for unencrypted, interpretable reasoning chains because the judge must read intermediate reasoning to localize the decisive step. Frontier deployments increasingly summarize, encrypt, or discard those chains. How much attribution accuracy survives when the decisive step's reasoning is not in the trace — and is the loss concentrated in the step metric or the error-mode metric? Answerable today by re-running the benchmark with reasoning content stripped or replaced by provider summaries.

Sources#

• Who&When Pro: Can LLMs Really Attribute Failures in AI Agents? — Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu & Huazheng Wang (Penn State / AG2ai / MBZUAI / Oregon State / Mathos AI), WHO&WHEN PRO: Can LLMs Really Attribute Failures in AI Agents?, arXiv 2607.09996, 2026-07-10, empirical, 36pp / 11 tables / 34 figures. Sections used: §1 and §2.1 (motivation, the text-only limitation of every prior benchmark, Table 1's comparison), §3 (decisive-error formulation, warm-start pipeline, injection-step selection, post-hoc filtering, Table 2 statistics), §3.5 + Table 3 (human review), §4.1–4.2 + Table 4 (metrics and main results; the length and modality analyses quoted from the Figure 4 prose), §4.3 + Table 6 (protocol comparison), §4.4 (cost frontier), §4.5 + App. B (the w/ G split and both paired case studies), App. C + Table 5 (the 18-mode taxonomy), App. D.4 (Step@k), App. F.2 (base agent models and the unencrypted-reasoning requirement).
• Tables verified. Table 2, Table 3 and Table 4 were checked against the PDF and parse clean. Table 6 was verified arithmetically rather than trusted: its all-at-once and step-by-step Step columns mean to 63.18 and 55.58, reproducing the prose's +7.6pp exactly, and all-at-once is strictly highest on Joint for exactly eight of ten models as stated (tied with binary search on GPT-5.4, beaten by step-by-step on GLM-5). All figures quoted from prose or captions, per the table-reconciliation rule.
• Parse warnings. (1) The ingest checker's warn on Table 1 is a false positive — the paper's own column header is literally "# Benchmark", which collided with the row-label heuristic; every Table 1 value matches the PDF and the table is safe to cite. (2) Table 7 (App. F.1, agent-framework → benchmark map) is a genuine collapse and shift the checker missed, and is not cited as parsed anywhere on this page; the verified mapping is PixelCraft→ChartQAPro/CharXiv/EvoChart, Multi-Agent Debate→GPQA/MATH/MMLU-Pro/SciBench/MMMU-Pro, DyLAN→GPQA/SciBench, MacNet→BigCodeBench/HumanEval/LiveCodeBench-Pro, MetaGPT→BigCodeBench/LiveCodeBench-Pro, MathChat→MATH/MMLU-Pro/SciBench, Magentic-One→GAIA/SimpleQA-Verified, ALF-Agent→ALFWorld, EVA→LVBench. 2026-09-07: Table 7 was hand-rebuilt in the raw, one framework per row against the PDF (15 rows; the four GUI rows AgentOccam→WebVoyager, OpenAI CUA→OSWorld, CoAct→OSWorld, WebVoyager→WebVoyager were already intact); Table 5 was rebuilt later the same day — see (3). (3) Table 5 (the error taxonomy) also carries an uncaught collapse+shift — each family's sub-rows merge into one grid row and the family label slides into the Description column — but the taxonomy is recoverable and prose-confirmed: six families totalling exactly the stated 18 modes (perception 2, reasoning 4, planning 2, action 5, verification 2, coordination 3). Only the family/mode counts are used here, never a per-row mapping. 2026-09-07: both repaired in the raw against the PDF — Table 1's two header levels merged into one header row so the # Benchmark cell no longer sits in a data row, and Table 5 rebuilt one mode per row (18 rows, six families, all mode names and descriptions checked in page order); verify.py now reports ok on the raw.
• One bibliography repair (Højmark-Bertelsen) was made at ingest.
• An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures — Vikas Pahuja, Jonathan Brokman, Omer Hofman, Tamir Nizri, Daniel Vishna, Seraphina Goldfarb-Tarrant, Kelly Marchisio, Hisashi Kojima & Roman Vainshtein (Fujitsu Research of Europe / Cohere / Fujitsu Research), An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures, arXiv 2608.03735, 2026-08-04, single version, empirical, 22pp / 12 tables / 7 figures. Sections used: §3.1 (the five grounding-failure categories), §3.2 (the plan-extraction filter, saturation criterion, and the deliberately disjoint development configuration — Qwen2.5-32B-Instruct and Cohere Aya-Expanse-32B inside Open Deep Research, none of them in the quantitative experiments), §3.3 + App. B (the TART schema, its closed vocabularies, and the no-translate/no-answer/no-plan instruction), §4 (OWL setup, 165 tasks per language, o3-mini fixed as coordinator and reasoning-coding agent), §5 (all headline gains, quoted from prose), §5.1 (the cumulative ablation), §5.2 (the resource gradient and the aggregate operation-grounding shares), App. E.2 (the Level-3 plateau diagnostic, Tables 6 and 7), App. G (the forced-single-primary-category convention, the judge model — Claude Opus 4.8 via AWS Bedrock, max_tokens 8192, applied only to baseline failed samples — and the other rule that captures target-versus-English query divergence), App. G.3 + Table 10 (per-category counts), App. G.4 + Table 11 (human verification), App. H + Table 12 (Common Crawl shares), and Limitations.
• COI, and it is structural on two axes. Six of the nine authors (Hofman, Brokman, Pahuja, Marchisio, Goldfarb-Tarrant, Vainshtein) are also authors of GAIA-MAPS (Hofman et al., EACL 2026 Findings), the benchmark this paper evaluates on and then extends by six machine-translated languages with no external check on translation quality. And two authors are at Cohere, whose Aya-Expanse-32B is one of the two derivation models — mitigated by that model appearing only in the development set that is discarded by design, with none of the three evaluated backbones being Cohere's. Neither COI reaches the within-paper comparison, which gives baseline and TART identical translations.
• Tables verified. Tables 5, 8, 9, 10 and 12 — the load-bearing ones — were all reconciled at ingest and re-checked at compile. Table 5 reproduces every per-language figure quoted here from its Run-1 columns (Avg 24.85 → 30.46 / +5.61; de +9.09, hi +9.69, ar +7.87, yo +10.91, ig +7.88) and the abstract's "+5.6 points". Table 10 was verified arithmetically rather than trusted: all 21 rows sum to their own printed Totals, all six category columns sum to their own grand Totals (857 / 917 / 915), and the prose's operation-grounding shares reproduce exactly (206/857 = 24.0%, 335/917 = 36.5%, 276/915 = 30.2%) — an internal check a collapse or shift would break. Table 6's taxonomy-covered percentages reproduce as (Total − Other)/Total for all three levels (60.4 / 55.2 / 55.2%) and Table 11's category n sums to its own Overall row (117). One confirmed collapse: Table 1 (App. B.1 inference config) merged its last two rows into one welded grid row (Mistral Large 3 Qwen3-VL-235B-A22B | MultiTAT MultiTAT | temp.: 0 temp.: 0), repaired at ingest against pdftotext -f 14 -layout; it is an inference-config table and nothing here rests on it. Caption-side alternation is real in this document — captions sit above Tables 1–7, 10 and 11 and below Tables 8, 9 and 12 — so every pairing was read by direct adjacency rather than by position. One rounding drift, non-load-bearing: Table 9's own Qwen3-VL average gain prints +3.04 while the prose two sentences later says "+3.1".
• canary-recall did not run and must not be read as a pass — 0 qualifying unique tokens, because the paper restates its headline numbers verbatim across abstract, introduction and §5, which defeats the exactly-once sampling heuristic. The manual substitutes were the two arithmetic self-checks above plus prose reconciliation of every quoted figure.
• One prose figure that fails to reconcile and is therefore not cited anywhere. §5.2 states the Other share "drops substantially from English to the lowest-resource languages (Yoruba and Igbo) — by 18% under Mistral-Large-3 and 16% under Qwen3-VL-235B." Table 10 gives Mistral 73/115 = 63.5% in English against 8/152 = 5.3% (Yoruba) and 4/153 = 2.6% (Igbo), and Qwen 93/115 = 80.9% against 8.8% and 6.0% — drops of roughly 58–75 points, not 16–18, under every denominator tried (failures, tasks, model-averaged pooling). The direction is supported by a much larger margin than the paper claims for it, so nothing on this page needs the number; the mismatch is recorded because a reader quoting "18%" would be quoting a quantity the paper never defines.
• Figures: 2 of 7 opened under the image two-pass rule, and one is corrected against the scouting note. Figure 1 is not the taxonomy diagram — it is a two-panel results figure: (a) seven per-language failure-share pies on a log Common-Crawl axis, (b) an OWL baseline-vs-TART bar chart. The taxonomy diagram is Figure 2 ("Planner-specific multilingual grounding-failure taxonomy" plus a worked Igbo entity-grounding example). Figure 1(a) confirms the gradient's shape qualitatively (grey Other collapsing, pink operation grounding dominating Yoruba and Igbo) and is the only figure content used here. Figure 1(b) mixes denominators and is not cited: its GPT-5-mini bar plots the eleven-language average (24.8 → 30.5) beside seven-language averages for Qwen (19.9 → 23.5) and Mistral (19.6 → 25.5), while §5's cross-model-comparable GPT-5-mini figure on the shared seven languages is 24.8 → 32.2 (+7.4) — the caption says only "averaged over the languages." The error is conservative on the gain, and the base values coincide at 24.8 by arithmetic accident. Figure 4 was deliberately not used: its bar order is not its Common-Crawl label order (the GPT-5-mini panel plots Swahili before Kyrgyz and Nyanja ninth despite Nyanja being the lowest-resource language in Table 12), so any per-language value read off it risks misattribution; every per-language number here comes from Table 5 or prose instead.
• One internal inconsistency in the human-validation accounting, logged rather than resolved. Coverage is stated three incompatible ways: §5.2 says "6 annotators verified 122 samples… ~14% of failures from Mistral-Large-3"; App. G.4 says "each of the seven annotators is assigned 40 samples, giving 280 annotated samples out of 917… approximately 30%," then reports "117 decisive human verifications" by "5 different annotators"; and Table 11's caption excludes "six uncertain annotations" (117 + 6 = 123, not 122). The only internally consistent object is Table 11 itself, whose per-category n sums to 117 and whose 104/117 reproduces the stated 88.9%. The κ = 0.860 and macro-F1 = 0.906 figures agree across both statements and are the ones cited.
• TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents — TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents, Tao, Peng, Yao, Ge, Cheng, Wang, Gao, Li (UW–Madison + Microsoft Research), arXiv 2607.13988, v1 2026-07-15, empirical. Cited here only for §3.2–3.3 (the frozen-reference log-ratio value and its per-turn TD differences) and §A.5 (the V ≥ −0.3 answer-secured-prefix definition and the ten annotated credit traces — five successes localizing the winning turn, five failures losing a secured prefix to one diagnostic call). Quoted from prose and the printed credit sequences; no table on this page's topic. Full treatment and all parse notes on Turn-Level Credit Assignment.