The Open-Weight Frontier Gap

Sources#

Summary#

Table 4 of the Gemma 4 report is a snapshot of the open-weight landscape as of June 19, 2026, measured on Arena Text — blind side-by-side human preference, Elo-rated. It is the only externally adjudicated number in a document otherwise full of self-reported benchmarks, and it is more informative than any of them.

Rank	Model	Elo	Open	Type	Params / active
1	Claude Fable 5	1508	no	—	—
~15	GLM 5.1	1475	yes	MoE	744B / 40B
29	MiMo V2.5 Pro	1466	yes	MoE	1T / 42B
34	Kimi K2.6	1460	yes	MoE	1T / 32B
36	DeepSeek V4 Pro Thinking	1458	yes	MoE	1.6T / 49B
43	Gemma 4 31B	1451	yes	Dense	31B
57	Qwen 3.5 397B-A17B	1444	yes	MoE	397B / 17B
61	Gemma 4 26B-A4B	1438	yes	MoE	26B / 4B
157	Gemma 3 27B	1366	yes	Dense	27B

Three readings#

The gap is small in Elo and enormous in parameters. The best open model trails the best closed model by 33 Elo; Gemma 4 31B trails it by 57. But GLM 5.1 buys its 24-point lead over Gemma with 744 billion parameters against 31 billion — a 24× ratio, and it activates 40B per token, more than Gemma's entire dense model. "Open weights have nearly caught up" and "open weights at the frontier require a datacenter" are both true, and they are the same sentence read from different ends.

Frontier-open means MoE; Gemma is not playing that game. Every open model above Gemma 4 31B is a Mixture-of-Experts in the 397B–1.6T range. Gemma's own summary — "the leading dense open model on the leaderboard" — is a precisely scoped claim, and the scoping is the point. The family's stated target is "varied hardware environments" and "edge deployment." It competes on inference efficiency, where a 0.8 GB quantized E2B is a category the 1.6T models cannot enter at all. Two open-weight strategies have separated: approach the frontier with sparsity, and approach the device with efficiency.

And the sparsity side now has a documented training method. The GLM 5.1 at the top of this table (744B/40B) is the immediate predecessor of GLM-5.2 (750B-A40B), which the SAO paper reports training with asynchronous single-rollout RL. So the corpus now has both ends of the frontier-open MoE story: where these models land (this page) and how they are trained to get there (Asynchronous RL for LLMs). The same paper's Table 1 also shows GLM-4.7 beating GPT-5 High and Claude-Sonnet-4.5 on three of four math benchmarks — the capability-side counterexample to Gemma's efficiency-side positioning, on measured benchmarks rather than Arena Elo.

DeepMind's MoE loses to DeepMind's dense model. Gemma 4 26B-A4B (Elo 1438, rank 61) sits 13 Elo below Gemma 4 31B (1451, rank 43), on human preference, from the same lab in the same release — while every larger open model in the table proves MoE scales. On static benchmarks the MoE is close to the dense model (MMLU Pro 82.6 vs 85.2, AIME 88.3 vs 89.2) and sometimes ahead (τ²-airline 76.0 vs 75.0), but human raters prefer the dense 31B. The report does not remark on this. If the effect is real, it suggests sparsity's returns arrive at scales far above 26B, or that active-parameter count (3.8B) governs the qualities Arena raters respond to.

Why the human-preference number is the trustworthy one#

The rest of the report is Google measuring Google. Arena is blind, third-party, human-rated, and confidence-intervalled (Gemma's ±8). It also disagrees with the static benchmarks in a legible way: Gemma 4 31B is claimed to rival "larger, frontier open models," and on Arena it does — 1451 against DeepSeek V4 Pro's 1456, inside the combined error bars. That is a real result, and it is real precisely because Google didn't grade it.

The same table quietly cites Claude Fable 5 at rank 1 — a third-party corroboration of a model whose own wiki page rests entirely on Anthropic's vendor-claim announcement, and whose head-to-head benchmark table was published only as an untranscribed image. A competitor's leaderboard is better evidence for Fable 5's standing than Fable 5's launch post.

What the table does not control for#

Arena Elo carries no compute budget. Gemma 4's entries are thinking-mode models; the table doesn't say at what thinking budget they were served, nor what the closed models spent. Per Compute-Controlled Benchmarking, a preference score without a budget has the same defect as a benchmark score without one — it just hides it behind human judgment instead of a number. The 33-Elo open-versus-closed gap could be a capability gap, an inference-spend gap, or both.

Connections#

Gemma 4 — the source of the table; the leading dense open model
Claude Fable 5 — rank 1, and cited here by a competitor rather than by its vendor
Inference Efficiency as Capability — the axis Gemma competes on instead of scale
Compute-Controlled Benchmarking — an Elo score without a budget is still a score without a budget
Open-Weight Elicitation Irreversibility — what "open" costs, once these models carry a thinking mode
Jagged Intelligence (Ghosts, Not Animals) — the aggregate Elo hides that small Gemmas beat Gemma 3 27B on reasoning and lose on knowledge
Large-Scale Test-Time Compute — the unnamed variable underneath every cell of the table
Encoder-Free Early Fusion — one of the levers that lets a 31B dense model contend at all
Responsible Scaling Policy Evaluations — why the open-weight safety argument here is structural: Gemma 4 sits at rank 43, nowhere near a risk threshold
Task Time-Horizon Scaling — Arena scores chat preference; whether the open/closed gap survives on long-horizon agentic work is a different measurement
Google DeepMind — publishes the table, and places itself 43rd on it
GLM (Z.AI) — GLM 5.1 (top of the table) and its SAO-trained successor GLM-5.2: the capability-side open-weight strategy this page contrasts with Gemma's efficiency-side one
Single-Rollout Optimization — the RL method behind the GLM MoE line's continued frontier presence; the training-side complement to this landing-place snapshot

Open questions#

Is the dense-beats-MoE result at 26B robust, or an artifact of one Arena snapshot with ±8 error bars on both models? (The two intervals overlap: 1451±8 and 1438±8.)
The open MoE giants (GLM, DeepSeek, Kimi, MiMo, Qwen) are overwhelmingly Chinese-lab releases. Gemma is the Western open-weight entry and it targets the device, not the frontier. Is that a strategic choice or a capability constraint?
Arena measures preference on chat. Does the 33-Elo open/closed gap widen or collapse on long-horizon agentic work, where time-horizon rather than response quality governs?

Sources#

Gemma 4 Technical Report — Table 4, Arena Text leaderboard as of 2026-06-19 (empirical; third-party human ratings, unlike the rest of the report's self-reported benchmarks); §4.1 human evaluation
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning — GLM-5.2 (750B-A40B) as SAO's deployment target; GLM-4.7 on Table 1 (empirical, first-party to the GLM lab)