Epoch Automation Reports

Sources#

• Can AI automate Epoch?
• Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI

Summary#

Epoch AI's Can AI automate Epoch? (Can AI automate Epoch?, Kelly Hong and Greg Burnham, 2026-10-08, empirical) launches a recurring report series in which models attempt Epoch's own work and are graded against the standard Epoch expects of its employees. The aim is to measure what benchmarks of "easily verifiable tasks that capture only a subset of real work" leave out. The answer for October 2026 is that no model can yet do an Epoch job end to end. The best closed models fail on judgment, not on execution, and the open-weight models also fail on execution.

The page records it as an instrument, not a single result, because Epoch says it will "add results as new models are released." Later editions belong here.

The design#

Tasks. There are 11 tasks in five categories, all taken from what Epoch staff actually do:

A fourth graphic-design task, restyling a Matplotlib chart into Epoch house style from the website repository, was dropped from the suite because Fable 5.1 produced Epoch-quality output. Epoch says it now largely automates that step in its own work.

Context at new-hire parity. The stated rule is to give each model "as much context… as we would to a new hire": full permissions in its own workspace and Google account, and tool access the job requires (a Figma integration plus a Figma file holding months of prior designs, components and guidelines; Google Drive; satellite-imagery ordering sites). This is the axis on which the series differs from GDPval, which writes all needed context into the prompt and excludes proprietary tools. It also differs from the Remote Labor Index, which keeps tasks self-contained, and from AutomationBench, which runs simulated environments scored by programmatic checks such as "was this email sent?". The authors name CRUX as the closest prior work but say it tests one messy long-horizon task per evaluation, not a suite of everyday ones.

Each model in its own harness, at maximum effort.

Grading. Each category has a rubric built with Epoch staff expert in that task and refined from an initial qualitative pass that turned observed failure patterns into criteria. Rubrics mix objective items ("pre-chart and post-chart paragraphs each under 150 words") with subjective ones ("title is clearly communicated"). One human grader scores every output, and each model runs each task once. The authors also read trajectories, not only outputs, and say the qualitative findings are "more informative than these exact numbers."

The scores, approximately#

Figure 1 is a bar chart of average score across the five categories, where 100% means meeting the standard Epoch expects of its employees. The chart prints no value labels, so the figures below are read from bar lengths against the axis and carry about ±1 point of reading error on top of the study's own noise:

Two cautions. The bottom bar is a Flash-tier Google model, not Google's frontier tier, so it does not place Google against the other labs. And the gaps between neighbouring bars (Qwen vs Kimi, about a point) are well inside what one run and one grader can resolve.

What the models got right#

The closed leaders' advantage is reliability on the well-defined parts: coding and computational analysis were "consistently accurate," and computer use was "far more capable than we expected." Earlier in 2026, frontier closed models failed to port an article to Substack, and the newer ones succeeded. No closed model made a factual error in its Data Insight, and every closed model's graphic was factually accurate.

Five failure patterns#

1. Open-weight models fail where closed ones are reliable. Kimi K3 built an entire arXiv Data Insight on a filtering error: it counted AI mentions across all arXiv papers rather than physics papers, "and never checked whether its filter worked." Its wealth-redistribution diagram drew returns flowing from people into capital assets, the reverse of the prompt and of its own caption, while it spent repeated rounds on minor design details. Epoch's reading: "This capability gap is larger than existing benchmark scores suggest." Kimi K3 scores 158 on the Epoch Capabilities Index, roughly level with Grok 4.6, "despite Kimi K3 struggling on basic tasks that Grok 4.6 handles more reliably." The ECI is Epoch's own index, so here the evaluator reports a limit of its own instrument. On how this bears on the open/closed question, see The Open-Weight Frontier Gap.

2. Implicit standards are not induced from examples. Asked for a simple diagram of "constant elicitation understates progress relative to max elicitation", with the Gradient Updates archive and the full Figma file as references, Fable 5.1 produced a dense graphic where an Epoch designer drew a simple chart. It copied "A"/"B" badges from one complex reference, where they earn their place because each concept recurs across the diagram, and used them for one-off labels. GPT-6 Astra, given every published Data Insight (which consistently target a general AI audience), chose a physics-niche topic for the arXiv task. Epoch later published the general-audience version itself, on math papers. The context was supplied, and the convention still was not learned. That is the finding with the widest reach; see Context Advantage, Not Taste.

3. Good directions, unexecutable experiments. "It's not difficult to identify a valuable general direction." GPT-6 Astra's research question (does an agent that fails to improve with practice fail to collect evidence or fail to use evidence it has?) is called genuinely promising. Its design left its critical pieces ("matched externally selected experience", "simulator-derived or restricted-policy reference bounds") undefined, so nobody could say how to run it.

4. Setup artefacts reported as findings. In the pilot, Astra gave the tested models a 4,096-token limit for choosing inputs, which cut off 61 of 280 responses before they named any input. The cut-off responses still counted as attempts. Astra noticed the problem, raised the budget, and scores recovered. It then wrote the result up as "sensitivity to the acquisition budget," framing a truncation bug as a property of the models it was studying. Epoch does not penalise setup mistakes, which are common in research. What it penalises is promoting the artefact to a key finding. It is the familiar evaluator's error of reading a benchmark artefact as a capability, made here by the model acting as the experimenter.

5. Excess density, and convergence. Fable 5.1 in particular produced information-dense summaries and verbose captions. Its own subagent's self-critique pushed for longer captions ("Descriptions are thin paraphrases…"), against Epoch's principle of communicating "with as little detail as possible." Across vendors, outputs converged: on the polling Data Insight, Fable 5.1, Gemini 3.8 Flash and Grok 4.6 chose the same topic (3 of 6), and Kimi K3 and Qwen 3.8 Max converged on the research-design task despite nine broad directions to choose from. This is the cross-vendor convergence that Agent Behavioral Homogeneity records, here on open-ended creative choices rather than on errors.

Evidence note#

empirical, with the authors' own caveats, which they state plainly:

• n = 1 per model per task, one grader. Scores "are therefore noisy and may shift with more runs," and "could change with different rubric item definitions or different graders." No inter-rater figure exists.
• Epoch-specific. The tasks and the standard are Epoch's. The authors disclaim any general measure of work automation.
• Harness confound. Each model ran in its own vendor's harness, and the authors name one plausible advantage: Codex's built-in browser extension against Qwen Code's Chrome DevTools MCP. This is not a small caveat for the open/closed comparison. Moonshot's own card reports K3 scoring higher under Claude Code than under Kimi Code, and Cline moved K3 11.3pp on Terminal-Bench from scaffold changes alone (Kimi (Moonshot AI)). So part of K3's shortfall here may belong to Kimi Code. Harness Tax: Coding-Agent Cost Multiplies Across Harnesses While Success Barely Moves finds harness choice moves coding success only ±2–5%, but that is on public coding benchmarks, not tool-heavy open-ended work.
• Interest. Epoch is grading whether AI can replace Epoch staff, by Epoch's standards, and the headline ("cannot yet replace workers, at least not at Epoch") is the one that keeps the grader employed. Against that, the most quotable finding cuts against Epoch's own product: its ECI fails to separate K3 from Grok 4.6. No model vendor was involved.
• Figure values are pixel-read, as above. Every other figure in the post is an example output with no score.

Where it sits among work-automation measures#

Two rulers already in the corpus measure "real work":

• GDPval, the one-shot, fully specified, prompt-contained end. Its sharpest qualitative result was "the models struggled to figure out context" when context was removed. Epoch inverts the ablation: it adds context at new-hire level, and the failures persist.
• Agentic Elo boards (GDPval-AA, AA-Briefcase) on The Open-Weight Frontier Gap, which put K3 35–61 Elo behind Fable 5 on vendor-transcribed numbers.

The Epoch series is the third. It is the least scalable (human-graded, single-run) and the closest to an actual job, and it disagrees with the index rather than refining it. Economic Benchmark Construct Validity finds a single aggregate factor, largely tracking release date, explains most leaderboard variance. Two models level on such an aggregate and separated by ~7 points and a class of basic errors here is the kind of residual that audit says aggregate indices hide.

For AI R&D specifically, it is the non-lab counterpart to METR's predeployment reading of Opus 5.5 (AI R&D Autonomy Evaluation (AECI)). Both name the same residue, judgment over which experiment is informative and which result to believe, and Epoch documents it in a model's actual write-up rather than in a summary sentence. See Research Taste as the Human Bottleneck.

An Epoch researcher's prior, two weeks before the report. Asked whether Epoch could hand its Chinese job-postings taxonomy to a model and rerun it every quarter, JS Denain, who leads Epoch's Insights Team, split the job the same way the report does (Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI, 2026-09-22, practitioner-opinion). Classifying postings into coherent categories across companies is "pretty cool, interesting, and automatable". Choosing what in the data is worth reporting is not: "I feel pretty often that LLM's assessment of what are the top five most interesting things is not that great." The report's research-design failure is that judgment observed in a graded task.

Connections#

• The Open-Weight Frontier Gap — the fourth axis on that page's agentic-gap question, and the first where a non-vendor grader puts closed and open models on the same real tasks: the gap is wider than the ECI shows
• GDPval Benchmark — the prompt-contained end of work-automation evaluation; Epoch adds the context GDPval's ablation removed and the failures do not go away
• Research Taste as the Human Bottleneck — patterns 3 and 4 are the residue that page defines (which experiments are informative, which results to trust), observed in trajectories
• Context Advantage, Not Taste — pattern 2 is the closest thing in the corpus to supplying the context and checking whether the gap closes; it did not, on implicit conventions
• Agent Behavioral Homogeneity — 3 of 6 vendors' models picked the same topic from an open idea space
• Economic Benchmark Construct Validity — why an aggregate index can place K3 level with Grok 4.6 while real tasks separate them
• Harness Tax: Coding-Agent Cost Multiplies Across Harnesses While Success Barely Moves — the harness confound in the setup, and why the coding-benchmark bound on it may not transfer
• AI R&D Autonomy Evaluation (AECI) — the lab-side framework for the same automation question; the research-design task is a small outside instance of it
• The US–China Capability Lag Under a Compute Deficit — why the ECI-matched K3/Grok pair matters for the US–China lag: Denain's 6–8 months is read off the same index this report finds too generous to the open model
• Epoch AI — the author, and owner of the ECI that this report finds misses the gap
• Kimi (Moonshot AI) / GPT-6 Astra / Claude Fable 5 — the three models with entity pages here

Open Questions#

• Does Kimi K3's shortfall survive a harness swap? Running K3 in Claude Code or Codex on the same 11 tasks would split model from scaffold, and Moonshot's own card suggests the swap could raise it.
• Is the ranking stable across graders and runs? Epoch says it "may add multiple graders in the future"; the next edition with a second grader or repeated runs is the trigger.
• Is the ECI-vs-real-work divergence specific to K3, or do open-weight models generally score lower on real tasks than closed models at the same ECI? A second ECI-matched open/closed pair on the same suite would settle it. (2026-10-15, a candidate mechanism, not an answer: Denain (Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI) says an internal Epoch check found no significant contamination effect on the US–China ECI lag, and suggests instead that US labs optimize capabilities that "appear less in benchmarks". That would make the divergence general to the open/closed split rather than specific to K3.)

Sources#

• Can AI automate Epoch? — Kelly Hong & Greg Burnham (Epoch AI), "Can AI automate Epoch?", epoch.ai, 2026-10-08 (empirical; single run per model per task, single human grader, author-acknowledged). Task suite, harness/effort table, rubric method, the five failure patterns with their worked examples (Kimi K3 arXiv filtering and wealth-diagram errors; Fable 5.1 Model Performance Chart and captions; GPT-6 Astra's niche topic, underspecified design and 4,096-token / 61-of-280 truncation), the ECI 158 statement, and the limitations list. Figure 1 values are read from unlabelled bars (±1 point); cite them as approximate. The other 14 figures are example outputs, described in prose and not downloaded
• Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI — Lambert with JS Denain (Epoch AI), Interconnects podcast, 2026-09-22, practitioner-opinion. Cited for Denain's view on automating the job-postings analysis and for the relayed internal contamination check (not in raw/)