Inference Efficiency as Capability

Sources#

• CS329A Self-Improving AI Agents — Part 9: Future Research Areas
• Gemini 3.5 Flash-Lite Model Card
• Gemma 4 Technical Report
• Jeff Dean: The 1% Rule for Building in AI
• Kimi K3 Model Card
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

Summary#

Under Noam Brown's thesis — capability is a function of how much money you put into inference — a corollary follows that neither source states outright: anything that lowers the cost of a unit of inference raises capability at a fixed budget. Efficiency engineering stops being a footnote to the model and becomes a capability lever with the same units as scaling.

Gemma 4 (empirical) is the worked example. Its abstract sells "a leap in performance"; its substance is five distinct levers for making a token cheaper to produce. Read through Brown's frame, the KV-cache and quantization work is not plumbing beneath the capability story — it is the capability story, told from the denominator.

This page is also the wiki's first coverage of the deployment half of the stack. Everything else here concerns what models do; this concerns what it costs to let them do it.

The five levers (Gemma 4)#

1. KV cache — 37.5% off the global cache. Three compounding choices. A 5:1 ratio of local sliding-window to global self-attention layers (4:1 for the 2.3B model), so most layers never hold a full-context cache. p-RoPE with p = 0.25 on global layers — only a quarter of the dimensions get rotary encoding (RoPE frequency 1M global, 10k local). And the strange one: reuse keys as values in the global layers, values = keys, deleting the V projection outright (citing Kayyam et al. 2026, Do transformers need three projections?). E2B and E4B skip the keys-as-values trick and instead share the KV cache across layers at ratios of 20/35 and 18/42.

2. Quantization-aware training — models fall below a gigabyte. Two weight formats: mobile quantization (per-channel int2/int4 weights, int8 activations) and Q4_0 blockwise, chosen for what llama.cpp and consumer hardware actually run. Text-only footprint at 32k context:

Model	bf16	Quantized	+KV cache
E2B	4.6 GB	0.8 GB	+0.05
E4B	9.0 GB	2.3 GB	+0.14
12B	24.0 GB	7.65 GB	+0.28
31B	64.0 GB	19.2 GB	+1.10

QAT is applied to the encoders too: the 150M vision encoder at W8A8 halves forward-pass memory (400 → 200 MB) and cuts on-device latency 44% versus Gemma 3n; the audio encoder, quantized to 8-bit activations and per-layer-cluster {2,4,8}-bit weights, shrinks 78% on disk, 390 MB → 87 MB. A scalar scale per block bounds activation ranges so fp16 inference stays stable.

3. Speculative decoding, shipped as an artifact. Gemma 4 releases a multi-token-prediction drafter head with each model (76M for E2B up to 500M for the 31B): a separate embedder plus a 4-layer Transformer block (three local, one global attention layer) that cross-attends to the main model's KV cache. Because the drafter reads the main model's KVs rather than running its own prefill, it needs no MTP prefill and supports any draft length. For the small drafters, the final projection is replaced by a top-k over token clusters, shrinking the last matmul from d × 262,000 to d × 4,096 at a similar acceptance rate.

4. Mixture-of-Experts. 26B total, 3.8B active — the standard sparsity trade. Notably it is the weakest lever here on human preference; see The Open-Weight Frontier Gap.

5. Removing the encoders. The 12B's 550M vision encoder becomes a 35M matmul and its 305M audio conformer is deleted outright, "alleviating the need for separate encoders and reducing memory fragmentation." Motivated by memory, not accuracy — see Encoder-Free Early Fusion.

The same logic at the opposite pole: Kimi K3 (July 2026)#

Gemma 4 spends efficiency to fit a phone. Kimi K3 (vendor-claim) spends it to make a 2.8T-parameter open model runnable by anyone at all — the same denominator argument, inverted. Its levers are different in kind and all of them are load-bearing rather than optional:

• 3.7% activation sparsity. 104B active of 2.8T total, routing 16 of 896 experts per token plus 2 shared. That is the sparsest routing in this corpus by a wide margin (Inkling: 6 of 256). MoE was Gemma's weakest lever at 26B; at 2.8T it is the only thing that makes the parameter count meaningful, which is consistent with The Open-Weight Frontier Gap's reading that sparsity's returns arrive far above 26B.
• Hybrid attention: 69 Kimi Delta Attention layers to 24 Gated MLA. Structurally the same bet as Gemma's 5:1 local-to-global ratio and Inkling's 5:1 sliding-to-global — keep a minority of layers paying full-context attention cost — but with a linear-attention variant rather than a sliding window in the majority slot. Three labs, three mechanisms, one ratio class, all aimed at the KV cache. K3 buys a 1M-token context with it.
• MXFP4 weights / MXFP8 activations, quantization-aware from the SFT stage onward. Gemma applies QAT to reach sub-gigabyte checkpoints on consumer hardware; Moonshot applies it "for broad hardware compatibility" at the other end, where a 2.8T bf16 checkpoint is simply not deployable. Same technique, opposite constraint: one is fitting into RAM, the other into a fleet.

The headline is "an approximate 2.5× improvement in overall scaling efficiency over Kimi K2" — and it is the one number here that cannot be checked. The card gives no definition of the metric, no baseline curve, no ablation separating KDA from AttnRes from LatentMoE, and no cost axis. This is precisely the failure mode Compute-Controlled Benchmarking names, arriving from the efficiency side rather than the capability side: an efficiency claim without an axis is as underspecified as a benchmark score without a budget. Gemma 4's efficiency work is auditable (memory footprints, WER deltas, latency percentages) and is under-credited by its own grid; K3's is un-auditable and is the release's headline.

One cost lands on the caller rather than the server. K3 requires preserved thinking history — every prior assistant message must be passed back with reasoning_content intact — so a long agentic session carries its full reasoning trace in-context rather than discarding it. That is an efficiency transfer: cheaper for the model (no re-derivation), more expensive in the context window, and it interacts with the fact that K3 has no thinking-off mode.

The efficiency tier moves up-market: Gemini 3.5 Flash-Lite (July 2026)#

Gemma 4 and K3 are efficiency at the level of the architecture — levers inside the model that make a token cheaper to produce. DeepMind's Gemini 3.5 Flash-Lite card (2026-07-21, vendor-claim) is the first source here on efficiency as a product tier, and it moves in the direction the "efficiency is capability" frame does not predict: the successor to an efficiency-tier model is more expensive. Output price goes $1.50 → $2.50 per 1M tokens (+67%), input $0.25 → $0.30 (+20%), against a predecessor it otherwise dominates.

What the price buys, per DeepMind's own table (3.1 Flash-Lite → 3.5 Flash-Lite):

Benchmark	3.1 FL	3.5 FL	Δ	Relative gain	Clears the +67% output-price rise?
MLE-Bench	22.0%	39.2%	+17.2pp	+78%	yes
Terminal-bench 2.1	31.0%	54.0%	+23.0pp	+74%	yes
GDM-MRCR v2 (1M pointwise)	12.3%	21.3%	+9.0pp	+73%	yes
SWE-Bench Pro	38.3%	54.2%	+15.9pp	+42%	no
OSWorld-Verified	54.3%	74.0%	+19.7pp	+36%	no
GDM-MRCR v2 (128k avg)	60.1%	72.2%	+12.1pp	+20%	no
CharXiv (no tools)	73.2%	74.5%	+1.3pp	+2%	no
GDPVal-AA v2 (Elo)	642	1140	+498	— (Elo is interval-scaled; a ratio is meaningless)	n/a

The structural finding is the asymmetry, not the sign. The price rise is a single uniform number applied to every token. The capability gain is wildly non-uniform: a full tier on agentic work (terminal, ML engineering, computer use, knowledge-work Elo), noise on chart understanding. So within one vendor's cheapest tier, whether the new model is cheaper for you is now a workload question, not a spec-sheet question — the same buyer sees +78% value on an MLE-Bench-shaped workload and +2% on a CharXiv-shaped one, at the same +67% bill. "Flash-Lite" stopped naming a price point and started naming a latency-and-capability profile whose price floats.

Three caveats that the arithmetic above cannot escape, all of them the same caveat in different clothes:

• Percent-correct is not linear in value. A task class a model fails 62% of the time may be worth nothing at all, so 38.3 → 54.2 can be worth far more than 42%. The per-dollar column is a floor on the comparison, not the comparison.
• Input and output prices moved differently (+20% vs +67%). An input-heavy long-context workload — exactly where MRCR@128k's +12.1pp lands — faces something much closer to +20%, which flips that row's verdict. The single "price rose 67%" headline is itself an uncontrolled number.
• No token counts anywhere. Price per token is a rate; the bill is rate × tokens-per-task, and a model that reasons longer can be more expensive at a lower rate. This is Cost-per-Task Over Cost-per-Token's distinction arriving from the vendor side, and it is why the card's price rows are not compute control — see Compute-Controlled Benchmarking.

Cross-vendor, the card places 3.5 Flash-Lite at $0.30/$2.50 against GPT-5.4 mini ($0.75/$4.50) and Claude Haiku 4.5 ($1.00/$5.00): roughly parity on SWE-Bench Pro (54.2 vs 54.4), ahead on OSWorld-Verified (74.0 vs 72.1), far ahead on long context (MRCR@128k 72.2 vs 42.7), behind on Terminal-bench 2.1 (54.0 vs 59.2), CharXiv (74.5 vs 80.3) and GDPVal-AA Elo (1140 vs 1171) — at ~55% of the output price. Weight this as vendor-claim and note the comparator selection: both rivals are one or two generations behind the OpenAI and Anthropic models this corpus otherwise tracks.

The denominator below the denominator: energy, and why batching exists#

Every lever above is an architectural answer to a physical ratio that none of the three sources states. Jeff Dean states it (YC Startup School 2026, practitioner-opinion): a multiply costs on the order of one picojoule; bringing the operand in from HBM into the processor costs roughly 1000× that. "The unit that you measure everything is energy."

The consequence he draws is the one that reframes this page: batching is not a training convenience, it is amortization of that ratio. "If you didn't have that thousand X difference, then you wouldn't have to do batching… you avoid paying a thousand X slowdown, but instead pay a thousand X divided by batch size energy cost." Which means batch size and latency are structurally opposed — "for really low latency, batching is not very good" — and every KV-cache trick in the Gemma 4 and K3 sections above is, underneath, an attempt to move less data rather than to do less arithmetic. Keys-as-values deletes a projection whose cost is mostly the traffic; a 5:1 local-to-global ratio keeps most layers off the full-context cache; MXFP4 shrinks what has to cross the wire. The levers look heterogeneous and are all the same lever.

Dean's forward-looking claim is that the next specialization is inference silicon, on the same reasoning that produced the TPU: minimize data movement, and commit to a small fixed set of very low precisions rather than supporting many ("if you feel like you have a good answer for what kinds of precision you need, maybe just build that into the hardware and not much else"). His stated prize is "latency 50× better." Weight it as one architect's direction of travel — no roadmap, no measurement, and a standing COI (Google's Chief Scientist on Google's accelerator strategy).

Two things this adds that the model cards cannot. First, an upper bound on what architecture can buy: efficiency levers inside the model are competing against a hardware constant, and the constant is where the 1000× lives. Second, a reason the efficiency tier keeps moving up-market (the Flash-Lite section above): if low latency requires small batches and small batches forfeit the amortization, then latency is expensive in energy independent of capability — so a tier defined by latency has no reason to stay a tier defined by price.

The reverse term, measured: an efficiency lever that changes which content can influence the answer (August 2026)#

Every section above runs the page's thesis forward — cheaper tokens, more capability at a fixed budget. Ren, Sun, Yi & Yang (Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation, CUHK / University of Maryland, arXiv 2608.01676, 2026-08-03, empirical) are the corpus's first controlled measurement of the term running backwards, and they attack the axis nobody audits: sparse attention is judged on throughput, memory, and aggregate benchmark accuracy, and none of those can see a change in whose evidence the model is able to use.

Their framing of the gap is the sentence this page was missing: a selector that discards evidence blocks "does not just save compute. It changes which evidence can influence the answer, and how strongly."

What is actually measured — and it is not accuracy. For a fixed item with gold answer y_G and target wrong answer y_P, the route-specific margin is M_o = log p(y_P | x, o) − log p(y_G | x, o), and a content block's behavioural influence is how that margin moves when the block is admitted to the route rather than excluded. The primary quantity is a difference-in-differences, ∆ = H₃^sparse − H₃^dense: positive means sparse attention amplifies the probe more than dense does, negative means dense does. Read this as a content-influence proxy, not a task-accuracy benchmark — every number below is in logits of a two-way answer margin, and no accuracy figure appears anywhere in the paper.

The apparatus is the contribution more than any single number:

• Matched probe cards. Three query-bound cards of exactly 128 tokens — Gold (carries the correct label), Poison (a target wrong label), Benign (whitespace filler only, no legal label) — identical except at frozen index 64. Every pair therefore differs at one token, with length, padding, template and query binding held fixed, and a complete Latin schedule rotates every candidate identity through every legal label so label-token preference cannot leak into the content effect.
• Six-layout symmetry. All six permutations of the three cards, so every card occupies every slot twice and every ordered pair appears in both orders — slot and order confounds removed by construction rather than by covariate.
• Equal-cardinality forced routing. A shared K−1 core is frozen (protected scaffold blocks plus neutral filler, no G/P/B/R anchor admitted), then four counterfactual routes each add exactly one anchor, giving pairwise Hamming distance two. Each route triggers an independent full forward; no prefill KV state is reused across routes.
• A within-kernel dense baseline. The dense arm is the same model, items, prompts, layouts and scoring rule under patched eager attention at c = 0 — same forward path, only the compression ratio varies. At c = 0 the two measurement operations (route substitution vs token replacement) agree to a mean |H₃| discrepancy of 0.044 logits at 100% label agreement, which is 11×–49× smaller than main effects spanning 0.47–2.15 logits, and that 0.044 is retained as a quantified baseline in every estimate rather than assumed away.

Two mechanisms compete, and both have direct causal evidence.

Signal concentration — the selector prefers signal-bearing blocks, so retained content gains influence. Natural route receipts give G ≈ P ≫ B in all four model–task pairs (Qwen3/SCBench-KV: 0.758 / 0.759 / 0.470). The control that makes this non-trivial is that the Benign card holds 128 real tab-filler tokens, so a selector merely preferring non-empty blocks would not separate them.

Integration loss — discarding blocks severs cross-block attention paths, so the surviving signal weakens. Confirmed by ablation rather than inferred: masking the probe block to retain full self-attention but neither attend to nor be attended from any other block collapses its influence from 4.48 logits [4.10, 4.87] to exactly zero across all 1,536 units (max |H₃| < 10⁻⁴, mean 0.0000). With sparse partial connectivity at c = 0.25 sitting at 4.09 [3.92, 4.26], the three conditions form a monotonic dose–response from full to partial to zero cross-block connectivity.

The two coexist inside a single cell, which is the finding that most resists summarising: in Qwen3/SCBench-KV at c = 0.25 the selector admits the signal-bearing block at rate 0.994 and ∆ is nevertheless −0.31. Route-level preference does not predict output-level influence; they are different layers.

Compression ratio is the control variable. Holding everything else fixed and sweeping c ∈ {0.25, 0.50, 0.75}:

Model–task	c = 0.25	c = 0.50	c = 0.75 (95% CI)
Qwen3-8B / SCBench-KV	−0.31	+0.16	+0.93 [+0.81, +1.04]
Llama-3.1-8B / SCBench-KV	−0.73	−0.70	−0.60 [−0.64, −0.56]
Qwen3-8B / SciFact	+0.19	+0.08	−0.08 [−0.18, +0.02]
Llama-3.1-8B / SciFact	−0.53	−0.46	−0.29 [−0.41, −0.17]

Two sign reversals, running opposite ways, produced by moving one knob. Three of four cells shift toward more positive ∆ as compression rises; Qwen3/SciFact is the exception and descends. So "how much does sparsity change content influence" has no answer without naming the operating ratio, the model, and the task — which is Compute-Controlled Benchmarking's complaint arriving on the efficiency side with the axis actually plotted.

The consequence for anyone deploying an efficiency lever is that the pooled test designed to detect this returns nothing. Three preregistered pooled hypotheses, all testing for a uniform direction, are null after Holm correction at p = 0.995, 0.771, 0.541 — while the per-cell stratified family rejects 31 of 32 nulls. The null is produced by the effect, not by its absence; see Aggregate Cancellation for the general form and the detection moves.

Four caveats, two of them about the paper's own framing.

• The headline sentence overreaches the paper's strongest arm, and in a direction worth naming. The abstract and introduction sell the risk as a sparse model "amplifying misleading content and suppressing corrective content." Every measurement in the paper that uses real misleading passages runs the other way: all eight real-evidence ∆ estimates are negative (−2.15 to −0.36, i.e. dense amplifies the poison more), and all eight KVPress cells show compression reducing the poisoning rate relative to dense (−0.0115 to −0.1644, every interval below zero). The only sparse-amplifies cells anywhere in the document are thin one-token synthetic label probes. The defensible claim is that sparsification changes content influence heterogeneously in sign; "amplifies misleading content" is not what its own real-evidence arms measured.
• The paper contradicts itself in print, twice. §1 and §7.3 each assert "every cell moves toward more positive ∆ at higher compression" and then, within the same paragraph, name Qwen3/SciFact as moving the opposite way. Table 5, Appendix Table 8 and Figure 3 all agree that the correct count is three of four; the universal claim is an over-claim left standing next to its own correction.
• Scale. Every model is in the 7B–8B class across four open-weight architectures (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B, Qwen3-8B). No frontier-scale MoE is tested — which is precisely the regime where the levers on this page actually ship — and the authors name scaling as open. The block size is fixed at 128 and tied to their own BSFA-based operator, with selection at the final query row only.
• Power and provenance. Thin-probe cells use n = 64 items, rich real-evidence cells n = 16. The KVPress arm's outcome metric differs from ∆, so the authors state the convergence is "directional, not quantitative." And the acknowledgments credit Agon, an autonomous research system whose paper is authored by three of this paper's four authors — self-cited tooling on a preprint, not a disqualification but worth knowing.

What the page keeps from it: the first evidence that an efficiency lever's cost lands on content selectivity rather than on average accuracy, and a measurement protocol requiring only behavioural access — no weight modification, no architectural change, applicable to any model that exposes block identities. Also one operational finding the authors flag and nobody would guess: filler content is not neutral to the selector, so padding and formatting conventions belong inside the audit rather than outside it.

The metric this page never had: intelligence per watt (Stanford, late 2025)#

Every section above measures efficiency in whatever unit its source happened to use — gigabytes, percentages off a KV cache, dollars per million tokens, logits. Mirhoseini closes CS329A (CS329A Self-Improving AI Agents — Part 9: Future Research Areas, delivered 2025-12-05, practitioner-opinion) by proposing a single one, and the definition is deliberately blunt:

intelligence per watt = average task accuracy ÷ average power draw to solve the task.

A capability numerator over a physical denominator, which is exactly the ratio Dean's picojoule argument says everything reduces to. The work is joint with Christopher Ré and John Hennessy at Stanford; Mirhoseini is an author and says so in lecture (standing COI, disclosed — see the note at the end of this section).

The question it was built to answer is a routing question, not a chip question: essentially all inference traffic goes to cloud accelerators today, so what fraction of it could be served locally instead? Three trends are stacked to motivate it, all as of late 2025 and all slide-read through ASR:

• Demand is compounding faster than anything else in the stack. Google Cloud's serving compute up ~1200× over the preceding ~20 months; NVIDIA growing ~10× year over year; Google's processed-token count from 160 trillion (Feb of the prior year) to 1.3 quadrillion (October); an implied ~250 GW of data-centre demand. The transcript garbles the units on the last figure and the periods on the first — carried here as orders of magnitude, not as measurements.
• Most queries do not need a frontier model. From a large ChatGPT usage dataset, roughly 77% of requests are practical guidance, information seeking, or writing — categories the study reads as answerable by small models, with the caveat the lecture states itself: users escalate their asks as chatbots improve.
• Local memory grew ~126× since 2012. A laptop with ~100 GB of unified memory can hold a quantized version of a model that used to require a datacentre — the QAT work above, arriving as a premise rather than as a result.

The study. "Local" is operationalized as ≤20B active parameters; >20 local models (the transcript legibly names quantized families and Llama 3), both enterprise and consumer accelerators, ~1M queries drawn from chat traffic plus reasoning benchmarks (NaturalReasoning, MMLU-Pro, SuperGPQA), scored on accuracy, energy, latency and compute, with the full grid open-sourced.

Three findings, and the third is the one that generalizes:

1. Local models cover most of the traffic. Accuracy on the query mix rose 3.1× since 2023, reaching — per the source's own abstract, since the ASR drops the number mid-sentence — ~88.7% of single-turn chat and reasoning queries. Treat the 88.7% as attributed to the paper via the raw document's description rather than as heard in the lecture.
2. Consumer silicon is behind, and not by much. An Apple M4 Max delivers roughly 1.5× lower intelligence per watt than a B200 — attributed to design intent rather than to process: datacentre parts were built for this workload and laptop parts were not, because "while designing these chips, the understanding wasn't the LLMs are going to be running" on them.
3. The metric improved 5.3× in two years, and it decomposes. 3.1× from better models × 1.7× from better hardware. That decomposition is the first thing in this corpus that separates the two contributions to an efficiency gain on one axis, and it is why the metric is worth more than the study: it makes "efficiency improved" an auditable claim with attributable parts, which is precisely what Compute-Controlled Benchmarking complains is missing and what K3's unauditable 2.5× headline lacks.

The forward claims, flagged as prediction inside a practitioner-opinion source and attributed inline. Mirhoseini predicts (i) inference traffic shifts substantially from cloud to edge as both curves continue, requiring hybrid serving engines that route between local and cloud by query complexity; (ii) model architectures and kernels optimized for energy on local accelerators are an under-attended research area; and (iii) energy becomes "the most valuable resource we have going forward," with intelligence-per-watt-style metrics becoming mainstream optimization targets.

Graded against this wiki's 2026 sources, that forecast is half right and the half that missed is instructive.

• Right on the substrate. Gemma 4 (empirical, mid-2026) is the trend continuing exactly as predicted — E2B at 0.8 GB quantized, drafter heads, encoders deleted, explicitly targeted at llama.cpp and consumer hardware. The premise that a capable model fits on a laptop stopped being a projection.
• Unsettled on the routing shift, and the corpus leans mildly against it. Nothing here measures a cloud→local traffic share. What it does have is DeepMind's Gemini 3.5 Flash-Lite, where the efficiency tier itself moved up-market — output price +67% — which is not what a market being eaten from below looks like. The sharper problem is definitional: the study's unit is a single-turn chat or reasoning query, and by mid-2026 this corpus's workloads are agentic and multi-turn (terminal sessions, hundred-tool-call tasks, hours-long autonomous runs). A 20B local model answering 88.7% of single-turn queries says little about the traffic that actually grew.
• Right on energy as the physical unit, wrong so far on the metric. Dean makes the same argument independently eight months later (~1 pJ per multiply against ~1000× to move the operand; "the unit that you measure everything is energy"). But no 2026 model card, benchmark, or eval page in this wiki reports a per-watt figure. The axis the market actually adopted is price per token — and its critique, cost per task, is a dollar-denominated proxy for the same ratio. Energy is the constraint; money is the instrument, and the pass-through between them is nobody's published number.

COI, and it is the best-disclosed instance in the nine-lecture series. Mirhoseini introduces the work as "a recent work that we did, in collaboration with Professor Ré and Professor John Hennessy", and the inference-systems work she cites as future direction — Hydragen and Tokasaurus — as "my lab did some of this work". Compare the pattern on CS329A: Self-Improving AI Agents (Stanford): lectures 2 and 5 taught her lab's papers in the first person without naming the authorship. The difference tracks framing, not honesty — work presented as current lab activity gets named; work presented as a canonical citation in a survey slot gets a pronoun.

Why this is a capability claim, not an engineering claim#

Brown's argument runs: a $10 inference budget does one thing, $10,000 does much more. He also observes that per-generation cost drops 10–100×, which is what makes waiting for the next model rational (Latent Capability Overhang).

Gemma 4 is that cost drop, decomposed and attributed. A model that fits in 0.8 GB runs on a phone, where the binding budget is RAM rather than dollars; a 37.5% smaller KV cache means longer thinking traces fit at the same memory ceiling; a drafter head means each token of a reasoning trace arrives sooner. A thinking mode and an efficiency stack are the same project: a reasoning trace multiplies tokens per query, so shipping one in a model people run locally is only coherent if you have first made tokens cheap. The report ships both in the same release and does not connect them.

The inverse holds and is worth naming. If efficiency is capability, then a compute-controlled benchmark (Compute-Controlled Benchmarking) is the only honest way to compare an efficient model against a large one — Brown's GPT-5.5-versus-5.4 anecdote is precisely a case where the grid hid an efficiency gain. Gemma 4's own Table 5 then commits the error it should have caught.

What the bitter lesson does and doesn't touch#

The report simultaneously removes hand-engineered structure (encoders, per Sutton's logic) and adds a great deal of it (5:1 attention ratios, p = 0.25, keys-as-values, per-layer-cluster bit-widths, top-k cluster projection). This is not a contradiction. The Bitter Lesson is a claim about what the model learns — structure encoding human priors about the task becomes a ceiling. It says nothing about the arithmetic of running the network. Deployment engineering is exempt, and Gemma 4 is the cleanest illustration in the corpus: the encoders go because they encode a prior about modality; the KV-cache tricks stay because they encode nothing at all.

The same distinction sharpens Harness Shrinkage as Models Improve. Harnesses shrink because capability migrates inward. Inference-efficiency work does not shrink — it compounds, because there is no inward for it to migrate to.

Connections#

• Large-Scale Test-Time Compute — the root thesis; efficiency is its denominator
• CS329A: Self-Improving AI Agents (Stanford) — the course whose closing lecture proposes intelligence per watt, and whose whole first half is the reason the denominator matters: a self-improvement flywheel is an inference-heavy object, so the efficiency lecture is the bill for the other eight
• Rationale Bootstrapping (STaR) — the same lecture's other half, and the reason the two sit in one session: the flywheel's compute cost is what makes the per-watt question urgent rather than academic
• Compute-Controlled Benchmarking — you cannot see an efficiency gain on a grid that doesn't control for compute
• Latent Capability Overhang — the 10–100× per-generation cost drop is the mechanism, disaggregated here into its parts
• Encoder-Free Early Fusion — lever five, and the one with independent corroboration
• The Open-Weight Frontier Gap — efficiency is the axis on which a 31B dense model competes with 744B MoEs at all
• Open-Weight Elicitation Irreversibility — cheap inference is what makes unbounded elicitation of open weights affordable to anyone
• The Bitter Lesson — applies to learned structure, not to the arithmetic of serving it
• Asynchronous RL for LLMs — the training-side sibling: async RL is training-efficiency, the same "efficiency is capability" logic applied one loop earlier
• Effective Compute Scaling — efficiency gains enter the "effective compute" numerator the same way hardware and algorithms do
• Gemma 4 — the source model family
• Google DeepMind — the lab
• Noam Brown — the thesis this page inverts
• Kimi (Moonshot AI) — the same efficiency-is-capability logic at 2.8T: 16-of-896 routing, 69 KDA + 24 Gated MLA, MXFP4 QAT, and an unauditable 2.5×-scaling-efficiency headline
• Jeff Dean — the physical constant underneath every lever here (~1 pJ per multiply vs ~1000× to move the operand), the reason batching exists, and the argument that inference-specialized silicon is the next specialization
• Cost-per-Task Over Cost-per-Token — the buyer-side statement of the last caveat: price per token is a rate, and Flash-Lite's +67% is uninterpretable without tokens-per-task
• Aggregate Cancellation — the measurement failure this page's newest source demonstrates and the reason an efficiency-shaped capability cost is invisible: a metric averaged over strata reports the mean of a composition it does not record, so offsetting shifts sum to zero and three preregistered pooled tests read as non-events
• Tool-Output Pruning — the same compress-the-context lever one layer out, at the agent–environment boundary rather than inside attention, and the closest sibling on the one property both measure: the sign of a compression effect flips with the backbone. That page's head raises resolve rate on one MoE and lowers it on another with no mechanism offered; this page's audit finds sign heterogeneity across models, tasks and compression ratios by design, and identifies the ratio as the knob that selects it
• Context Window Smart Zone — the constraint sparse attention is deployed to relax, and the mechanism that complicates the trade: cheap long context bought by discarding blocks severs exactly the cross-block attention that carries a signal to the answer position (4.48 logits → 0 when isolated), so a bigger cheap window is not the same object as a bigger window
• Open Weights as Competitive Strategy — the geopolitical payoff of this page's premise. If cheap tokens are capability, then whoever supplies intelligence most cheaply sets the cost floor for every application built on top — which is the step Ng's competitiveness argument rests on and never argues for, treating it as given
• Intra-Trace Parallel Planning (SPRINT) — the same argument applied to the shape of a reasoning trace rather than to the serving stack: SPRINT trains a model to emit independent plans together so its own thinking leaves the critical path. Also a case this page's standard is not yet met — the reported axis is sequential tokens, and the wall clock is named as future work

Open Questions#

• Is there an efficiency-to-capability exchange rate? Brown asks whether high-budget performance can be predicted from cheap runs. The dual question: how many Elo points is a 37.5% KV-cache reduction worth, at a fixed dollar budget? Nobody reports this, because nobody plots the axis. Partially answered (2026-07-30): Gemini 3.5 Flash-Lite is the first artifact in this corpus to print price and capability in the same table for a model and its own predecessor, so a rate is computable in one direction — +67% output price buys +42% relative on SWE-Bench Pro, +78% on MLE-Bench, +2% on CharXiv. That answers "what did this generation's price rise buy" and still not "what is a KV-cache reduction worth": the axis is price-per-token rather than spend-per-task, the levers are not decomposed, and the exchange rate turns out to be workload-dependent rather than a single number. Further partially answered (2026-08-17, late-2025 source): Stanford's intelligence per watt is the first proposed unit for exactly this rate — accuracy over power — and its headline 5.3× two-year gain decomposes into 3.1× model and 1.7× hardware, which is the first attributed split of an efficiency gain in this corpus. Three reasons it still does not close the question: the denominator is watts and every deployment decision in this wiki is denominated in dollars, with no published pass-through between them; the numerator is accuracy on single-turn chat and reasoning queries, not the agentic multi-turn work these levers actually serve; and the decomposition is a two-factor split of an aggregate, not a per-lever rate — it still cannot say what a 37.5% KV-cache reduction is worth. practitioner-opinion, slide-read, and the lecturer is an author.
• values = keys deletes a third of attention's projections in the global layers with no reported loss. Which other projections are redundant, and does the redundancy grow with scale?
• Does an efficiency lever ever cost capability in a way a benchmark grid hides? Gemma 4's encoder-free 12B collapses on dense-text vision when tokens are cut — an efficiency-shaped regression invisible at max resolution. Partially answered (2026-08-12), and this is the first source to answer it in general form rather than as one vendor's anecdote: Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation (empirical) runs a dense-calibrated counterfactual audit of block-sparse attention and KV-cache eviction and shows the lever changing which supplied content can influence the answer — with a direct mechanism (isolating a probe block from cross-block attention collapses its influence from 4.48 logits to exactly zero across 1,536 units), a control variable (compression ratio, with two sign reversals across four model–task pairs), and three preregistered pooled tests returning null at p = 0.995 / 0.771 / 0.541 because opposite-signed cells cancel. So "yes, and here is the instrument" for the hiding half. Three reasons it stays open. The outcome is a logit-margin influence proxy, never task accuracy, so a hidden capability cost is inferred rather than measured. Every model is 7B–8B, so the regime where these levers actually ship is untested. And the direction is not the alarming one the paper's abstract advertises — all eight real-evidence estimates and all eight KV-eviction cells show compression reducing poison influence, with sparse amplification appearing only on one-token synthetic probes.

Sources#

• Gemma 4 Technical Report — §2.1–2.7 (attention ratios, p-RoPE, keys-as-values, QAT, MTP drafter, TPU infra), Tables 1–3 (empirical)
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — No Priors interview (2026-06-26): capability as a function of inference budget; per-generation cost drops of 10–100× (practitioner-opinion)
• Kimi K3 Model Card — §1 (the 2.5×-scaling-efficiency claim, LatentMoE sparsity), §2 spec table (16-of-896 routing, 69 KDA + 24 Gated MLA, 1M context), §4 (MXFP4/MXFP8 QAT from SFT onward), §6 (preserved-thinking-history requirement) (vendor-claim)
• Jeff Dean: The 1% Rule for Building in AI — Jeff Dean with Diana Hu, YC Startup School 2026 (2026-07-30, practitioner-opinion): §"Why AI Is Really an Energy Problem" — the ~1 pJ-per-multiply vs ~1000×-per-data-move ratio, batching as amortization of it (1000/batch_size), and inference-specialized silicon as the next specialization. COI: Google's Chief Scientist on Google's accelerator strategy; the picojoule ratio is stated by the interviewer and confirmed, not cited
• Gemini 3.5 Flash-Lite Model Card — Google DeepMind, 2026-07-21 (vendor-claim): the July 2026 evaluation table, whose first two rows are input and output price per 1M tokens for all four compared models; per-dollar arithmetic in the section above is computed from those rows, not stated by DeepMind. Machine extraction from a client-side-rendered page, numbers verified across three passes
• Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation — Xingyu Ren, Youran Sun, Chugang Yi & Haizhao Yang (CUHK / University of Maryland, arXiv 2608.01676, 2026-08-03, empirical, 24pp): §3 + Table 1 the BSFA route-replay validation across four architectures (13 of 16 cells significantly positive, zero significantly negative, zero identity-replay label flips); §4.1–4.5 the audit protocol (matched 128-token G/P/B cards differing at frozen index 64, six-layout symmetry, equal-cardinality forced routing over a frozen K−1 core, the c = 0 within-kernel dense arm and its 0.044-logit kernel-path baseline at 100% label agreement); §5.2 the three preregistered pooled nulls (p = 0.995, 0.771, 0.541) against 31 of 32 stratified rejections; §5.5 + Table 5 the three-ratio sweep and both sign reversals; §5.6 + Table 6 the routing receipts (G ≈ P ≫ B, Qwen3-SCB 0.758 / 0.759 / 0.470); §5.7 the cross-block ablation (4.48 → 0 across all 1,536 units, dose–response through 4.09 at c = 0.25); §6 + Table 7 the KVPress eviction arm; §8 Limitations. Table 5 was reconciled three independent ways before citing — against Appendix Table 8's fuller matrix digit-for-digit, against Figure 3, and against the §5.5 prose narration of all four rows; table-collapse and table-shift report zero occurrences and all 11 captions sit above their tables with no alternation. Figures 1, 2 and 3 viewed under the image two-pass rule; Figure 3 independently confirms every sweep value, and Figure 2 confirms Tables 3 and 4 (it also omits the marker for the Llama-SF c = 0.50 real-evidence row it labels — a defect in the paper's own plot, not the parse; the value is in Table 4). Cosmetic parse artifact: every em- and en-dash in the document is flattened to a plain hyphen, so quoted sentences here have had the dash restored. Two problems in the source itself, both recorded in the section above: the abstract's "amplifying misleading content" framing is contradicted in direction by all eight real-evidence ∆ estimates and all eight KVPress cells, and §1 and §7.3 each assert a universal compression trend one sentence before naming the cell that violates it. Provenance note: the acknowledgments credit Agon, an autonomous research system whose own paper (arXiv 2606.24177) is authored by three of this paper's four authors
• CS329A Self-Improving AI Agents — Part 9: Future Research Areas — Stanford CS329A lecture 9, Future Research Areas (Azalia Mirhoseini's half; delivered 2025-12-05, published 2026-08-03, practitioner-opinion with the forward-looking directions at prediction grade, YouTube auto-caption transcript, ~10.5k words). The intelligence-per-watt section above: the metric's definition, the demand and local-memory trends, the ~77% query-mix figure, the ≤20B-active-parameter definition of "local", the study's model / accelerator / workload / metric grid and its open-sourcing, the three findings and the 3.1 × 1.7 = 5.3× decomposition, and the four forward directions (hybrid local–cloud routing, energy-efficient architectures and kernels, high-throughput low-latency test-time-scaling infrastructure, energy as the scarce resource). The paper is not in raw/ and every figure is read off a slide by ASR: the ~250 GW demand figure has garbled units, the Google Cloud growth period renders as "12 20 months", and the headline coverage number is dropped mid-sentence by the transcript — the 88.7% carried above comes from the raw document's own description of the paper, not from the spoken audio, and is labelled as such at the point of use. COI, disclosed: Mirhoseini is an author of the work she presents ("a recent work that we did, in collaboration with Professor Ré and Professor John Hennessy"), and of the Hydragen and Tokasaurus systems she names as future direction ("my lab did some of this work")