Inference Efficiency as Capability

Sources#

Summary#

Under Noam Brown's thesis — capability is a function of how much money you put into inference — a corollary follows that neither source states outright: anything that lowers the cost of a unit of inference raises capability at a fixed budget. Efficiency engineering stops being a footnote to the model and becomes a capability lever with the same units as scaling.

Gemma 4 (empirical) is the worked example. Its abstract sells "a leap in performance"; its substance is five distinct levers for making a token cheaper to produce. Read through Brown's frame, the KV-cache and quantization work is not plumbing beneath the capability story — it is the capability story, told from the denominator.

This page is also the wiki's first coverage of the deployment half of the stack. Everything else here concerns what models do; this concerns what it costs to let them do it.

The five levers (Gemma 4)#

1. KV cache — 37.5% off the global cache. Three compounding choices. A 5:1 ratio of local sliding-window to global self-attention layers (4:1 for the 2.3B model), so most layers never hold a full-context cache. p-RoPE with p = 0.25 on global layers — only a quarter of the dimensions get rotary encoding (RoPE frequency 1M global, 10k local). And the strange one: reuse keys as values in the global layers, values = keys, deleting the V projection outright (citing Kayyam et al. 2026, Do transformers need three projections?). E2B and E4B skip the keys-as-values trick and instead share the KV cache across layers at ratios of 20/35 and 18/42.

2. Quantization-aware training — models fall below a gigabyte. Two weight formats: mobile quantization (per-channel int2/int4 weights, int8 activations) and Q4_0 blockwise, chosen for what llama.cpp and consumer hardware actually run. Text-only footprint at 32k context:

Model	bf16	Quantized	+KV cache
E2B	4.6 GB	0.8 GB	+0.05
E4B	9.0 GB	2.3 GB	+0.14
12B	24.0 GB	7.65 GB	+0.28
31B	64.0 GB	19.2 GB	+1.10

QAT is applied to the encoders too: the 150M vision encoder at W8A8 halves forward-pass memory (400 → 200 MB) and cuts on-device latency 44% versus Gemma 3n; the audio encoder, quantized to 8-bit activations and per-layer-cluster {2,4,8}-bit weights, shrinks 78% on disk, 390 MB → 87 MB. A scalar scale per block bounds activation ranges so fp16 inference stays stable.

3. Speculative decoding, shipped as an artifact. Gemma 4 releases a multi-token-prediction drafter head with each model (76M for E2B up to 500M for the 31B): a separate embedder plus a 4-layer Transformer block (three local, one global attention layer) that cross-attends to the main model's KV cache. Because the drafter reads the main model's KVs rather than running its own prefill, it needs no MTP prefill and supports any draft length. For the small drafters, the final projection is replaced by a top-k over token clusters, shrinking the last matmul from d × 262,000 to d × 4,096 at a similar acceptance rate.

4. Mixture-of-Experts. 26B total, 3.8B active — the standard sparsity trade. Notably it is the weakest lever here on human preference; see The Open-Weight Frontier Gap.

5. Removing the encoders. The 12B's 550M vision encoder becomes a 35M matmul and its 305M audio conformer is deleted outright, "alleviating the need for separate encoders and reducing memory fragmentation." Motivated by memory, not accuracy — see Encoder-Free Early Fusion.

The same logic at the opposite pole: Kimi K3 (July 2026)#

Gemma 4 spends efficiency to fit a phone. Kimi K3 (vendor-claim) spends it to make a 2.8T-parameter open model runnable by anyone at all — the same denominator argument, inverted. Its levers are different in kind and all of them are load-bearing rather than optional:

3.7% activation sparsity. 104B active of 2.8T total, routing 16 of 896 experts per token plus 2 shared. That is the sparsest routing in this corpus by a wide margin (Inkling: 6 of 256). MoE was Gemma's weakest lever at 26B; at 2.8T it is the only thing that makes the parameter count meaningful, which is consistent with The Open-Weight Frontier Gap's reading that sparsity's returns arrive far above 26B.
Hybrid attention: 69 Kimi Delta Attention layers to 24 Gated MLA. Structurally the same bet as Gemma's 5:1 local-to-global ratio and Inkling's 5:1 sliding-to-global — keep a minority of layers paying full-context attention cost — but with a linear-attention variant rather than a sliding window in the majority slot. Three labs, three mechanisms, one ratio class, all aimed at the KV cache. K3 buys a 1M-token context with it.
MXFP4 weights / MXFP8 activations, quantization-aware from the SFT stage onward. Gemma applies QAT to reach sub-gigabyte checkpoints on consumer hardware; Moonshot applies it "for broad hardware compatibility" at the other end, where a 2.8T bf16 checkpoint is simply not deployable. Same technique, opposite constraint: one is fitting into RAM, the other into a fleet.

The headline is "an approximate 2.5× improvement in overall scaling efficiency over Kimi K2" — and it is the one number here that cannot be checked. The card gives no definition of the metric, no baseline curve, no ablation separating KDA from AttnRes from LatentMoE, and no cost axis. This is precisely the failure mode Compute-Controlled Benchmarking names, arriving from the efficiency side rather than the capability side: an efficiency claim without an axis is as underspecified as a benchmark score without a budget. Gemma 4's efficiency work is auditable (memory footprints, WER deltas, latency percentages) and is under-credited by its own grid; K3's is un-auditable and is the release's headline.

One cost lands on the caller rather than the server. K3 requires preserved thinking history — every prior assistant message must be passed back with reasoning_content intact — so a long agentic session carries its full reasoning trace in-context rather than discarding it. That is an efficiency transfer: cheaper for the model (no re-derivation), more expensive in the context window, and it interacts with the fact that K3 has no thinking-off mode.

The efficiency tier moves up-market: Gemini 3.5 Flash-Lite (July 2026)#

Gemma 4 and K3 are efficiency at the level of the architecture — levers inside the model that make a token cheaper to produce. DeepMind's Gemini 3.5 Flash-Lite card (2026-07-21, vendor-claim) is the first source here on efficiency as a product tier, and it moves in the direction the "efficiency is capability" frame does not predict: the successor to an efficiency-tier model is more expensive. Output price goes $1.50 → $2.50 per 1M tokens (+67%), input $0.25 → $0.30 (+20%), against a predecessor it otherwise dominates.

What the price buys, per DeepMind's own table (3.1 Flash-Lite → 3.5 Flash-Lite):

Benchmark	3.1 FL	3.5 FL	Δ	Relative gain	Clears the +67% output-price rise?
MLE-Bench	22.0%	39.2%	+17.2pp	+78%	yes
Terminal-bench 2.1	31.0%	54.0%	+23.0pp	+74%	yes
GDM-MRCR v2 (1M pointwise)	12.3%	21.3%	+9.0pp	+73%	yes
SWE-Bench Pro	38.3%	54.2%	+15.9pp	+42%	no
OSWorld-Verified	54.3%	74.0%	+19.7pp	+36%	no
GDM-MRCR v2 (128k avg)	60.1%	72.2%	+12.1pp	+20%	no
CharXiv (no tools)	73.2%	74.5%	+1.3pp	+2%	no
GDPVal-AA v2 (Elo)	642	1140	+498	— (Elo is interval-scaled; a ratio is meaningless)	n/a

The structural finding is the asymmetry, not the sign. The price rise is a single uniform number applied to every token. The capability gain is wildly non-uniform: a full tier on agentic work (terminal, ML engineering, computer use, knowledge-work Elo), noise on chart understanding. So within one vendor's cheapest tier, whether the new model is cheaper for you is now a workload question, not a spec-sheet question — the same buyer sees +78% value on an MLE-Bench-shaped workload and +2% on a CharXiv-shaped one, at the same +67% bill. "Flash-Lite" stopped naming a price point and started naming a latency-and-capability profile whose price floats.

Three caveats that the arithmetic above cannot escape, all of them the same caveat in different clothes:

Percent-correct is not linear in value. A task class a model fails 62% of the time may be worth nothing at all, so 38.3 → 54.2 can be worth far more than 42%. The per-dollar column is a floor on the comparison, not the comparison.
Input and output prices moved differently (+20% vs +67%). An input-heavy long-context workload — exactly where MRCR@128k's +12.1pp lands — faces something much closer to +20%, which flips that row's verdict. The single "price rose 67%" headline is itself an uncontrolled number.
No token counts anywhere. Price per token is a rate; the bill is rate × tokens-per-task, and a model that reasons longer can be more expensive at a lower rate. This is Cost-per-Task Over Cost-per-Token's distinction arriving from the vendor side, and it is why the card's price rows are not compute control — see Compute-Controlled Benchmarking.

Cross-vendor, the card places 3.5 Flash-Lite at $0.30/$2.50 against GPT-5.4 mini ($0.75/$4.50) and Claude Haiku 4.5 ($1.00/$5.00): roughly parity on SWE-Bench Pro (54.2 vs 54.4), ahead on OSWorld-Verified (74.0 vs 72.1), far ahead on long context (MRCR@128k 72.2 vs 42.7), behind on Terminal-bench 2.1 (54.0 vs 59.2), CharXiv (74.5 vs 80.3) and GDPVal-AA Elo (1140 vs 1171) — at ~55% of the output price. Weight this as vendor-claim and note the comparator selection: both rivals are one or two generations behind the OpenAI and Anthropic models this corpus otherwise tracks.

The denominator below the denominator: energy, and why batching exists#

Every lever above is an architectural answer to a physical ratio that none of the three sources states. Jeff Dean states it (YC Startup School 2026, practitioner-opinion): a multiply costs on the order of one picojoule; bringing the operand in from HBM into the processor costs roughly 1000× that. "The unit that you measure everything is energy."

The consequence he draws is the one that reframes this page: batching is not a training convenience, it is amortization of that ratio. "If you didn't have that thousand X difference, then you wouldn't have to do batching… you avoid paying a thousand X slowdown, but instead pay a thousand X divided by batch size energy cost." Which means batch size and latency are structurally opposed — "for really low latency, batching is not very good" — and every KV-cache trick in the Gemma 4 and K3 sections above is, underneath, an attempt to move less data rather than to do less arithmetic. Keys-as-values deletes a projection whose cost is mostly the traffic; a 5:1 local-to-global ratio keeps most layers off the full-context cache; MXFP4 shrinks what has to cross the wire. The levers look heterogeneous and are all the same lever.

Dean's forward-looking claim is that the next specialization is inference silicon, on the same reasoning that produced the TPU: minimize data movement, and commit to a small fixed set of very low precisions rather than supporting many ("if you feel like you have a good answer for what kinds of precision you need, maybe just build that into the hardware and not much else"). His stated prize is "latency 50× better." Weight it as one architect's direction of travel — no roadmap, no measurement, and a standing COI (Google's Chief Scientist on Google's accelerator strategy).

Two things this adds that the model cards cannot. First, an upper bound on what architecture can buy: efficiency levers inside the model are competing against a hardware constant, and the constant is where the 1000× lives. Second, a reason the efficiency tier keeps moving up-market (the Flash-Lite section above): if low latency requires small batches and small batches forfeit the amortization, then latency is expensive in energy independent of capability — so a tier defined by latency has no reason to stay a tier defined by price.

Why this is a capability claim, not an engineering claim#

Brown's argument runs: a $10 inference budget does one thing, $10,000 does much more. He also observes that per-generation cost drops 10–100×, which is what makes waiting for the next model rational (Latent Capability Overhang).

Gemma 4 is that cost drop, decomposed and attributed. A model that fits in 0.8 GB runs on a phone, where the binding budget is RAM rather than dollars; a 37.5% smaller KV cache means longer thinking traces fit at the same memory ceiling; a drafter head means each token of a reasoning trace arrives sooner. A thinking mode and an efficiency stack are the same project: a reasoning trace multiplies tokens per query, so shipping one in a model people run locally is only coherent if you have first made tokens cheap. The report ships both in the same release and does not connect them.

The inverse holds and is worth naming. If efficiency is capability, then a compute-controlled benchmark (Compute-Controlled Benchmarking) is the only honest way to compare an efficient model against a large one — Brown's GPT-5.5-versus-5.4 anecdote is precisely a case where the grid hid an efficiency gain. Gemma 4's own Table 5 then commits the error it should have caught.

What the bitter lesson does and doesn't touch#

The report simultaneously removes hand-engineered structure (encoders, per Sutton's logic) and adds a great deal of it (5:1 attention ratios, p = 0.25, keys-as-values, per-layer-cluster bit-widths, top-k cluster projection). This is not a contradiction. The Bitter Lesson is a claim about what the model learns — structure encoding human priors about the task becomes a ceiling. It says nothing about the arithmetic of running the network. Deployment engineering is exempt, and Gemma 4 is the cleanest illustration in the corpus: the encoders go because they encode a prior about modality; the KV-cache tricks stay because they encode nothing at all.

The same distinction sharpens Harness Shrinkage as Models Improve. Harnesses shrink because capability migrates inward. Inference-efficiency work does not shrink — it compounds, because there is no inward for it to migrate to.

Connections#

Large-Scale Test-Time Compute — the root thesis; efficiency is its denominator
Compute-Controlled Benchmarking — you cannot see an efficiency gain on a grid that doesn't control for compute
Latent Capability Overhang — the 10–100× per-generation cost drop is the mechanism, disaggregated here into its parts
Encoder-Free Early Fusion — lever five, and the one with independent corroboration
The Open-Weight Frontier Gap — efficiency is the axis on which a 31B dense model competes with 744B MoEs at all
Open-Weight Elicitation Irreversibility — cheap inference is what makes unbounded elicitation of open weights affordable to anyone
The Bitter Lesson — applies to learned structure, not to the arithmetic of serving it
Asynchronous RL for LLMs — the training-side sibling: async RL is training-efficiency, the same "efficiency is capability" logic applied one loop earlier
Effective Compute Scaling — efficiency gains enter the "effective compute" numerator the same way hardware and algorithms do
Gemma 4 — the source model family
Google DeepMind — the lab
Noam Brown — the thesis this page inverts
Kimi (Moonshot AI) — the same efficiency-is-capability logic at 2.8T: 16-of-896 routing, 69 KDA + 24 Gated MLA, MXFP4 QAT, and an unauditable 2.5×-scaling-efficiency headline
Jeff Dean — the physical constant underneath every lever here (~1 pJ per multiply vs ~1000× to move the operand), the reason batching exists, and the argument that inference-specialized silicon is the next specialization
Cost-per-Task Over Cost-per-Token — the buyer-side statement of the last caveat: price per token is a rate, and Flash-Lite's +67% is uninterpretable without tokens-per-task

Open Questions#

Is there an efficiency-to-capability exchange rate? Brown asks whether high-budget performance can be predicted from cheap runs. The dual question: how many Elo points is a 37.5% KV-cache reduction worth, at a fixed dollar budget? Nobody reports this, because nobody plots the axis. Partially answered (2026-07-30): Gemini 3.5 Flash-Lite is the first artifact in this corpus to print price and capability in the same table for a model and its own predecessor, so a rate is computable in one direction — +67% output price buys +42% relative on SWE-Bench Pro, +78% on MLE-Bench, +2% on CharXiv. That answers "what did this generation's price rise buy" and still not "what is a KV-cache reduction worth": the axis is price-per-token rather than spend-per-task, the levers are not decomposed, and the exchange rate turns out to be workload-dependent rather than a single number.
values = keys deletes a third of attention's projections in the global layers with no reported loss. Which other projections are redundant, and does the redundancy grow with scale?
Does an efficiency lever ever cost capability in a way a benchmark grid hides? Gemma 4's encoder-free 12B collapses on dense-text vision when tokens are cut — an efficiency-shaped regression invisible at max resolution.

Sources#

Gemma 4 Technical Report — §2.1–2.7 (attention ratios, p-RoPE, keys-as-values, QAT, MTP drafter, TPU infra), Tables 1–3 (empirical)
Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — No Priors interview (2026-06-26): capability as a function of inference budget; per-generation cost drops of 10–100× (practitioner-opinion)
Kimi K3 Model Card — §1 (the 2.5×-scaling-efficiency claim, LatentMoE sparsity), §2 spec table (16-of-896 routing, 69 KDA + 24 Gated MLA, 1M context), §4 (MXFP4/MXFP8 QAT from SFT onward), §6 (preserved-thinking-history requirement) (vendor-claim)
Jeff Dean: The 1% Rule for Building in AI — Jeff Dean with Diana Hu, YC Startup School 2026 (2026-07-30, practitioner-opinion): §"Why AI Is Really an Energy Problem" — the ~1 pJ-per-multiply vs ~1000×-per-data-move ratio, batching as amortization of it (1000/batch_size), and inference-specialized silicon as the next specialization. COI: Google's Chief Scientist on Google's accelerator strategy; the picojoule ratio is stated by the interviewer and confirmed, not cited
Gemini 3.5 Flash-Lite Model Card — Google DeepMind, 2026-07-21 (vendor-claim): the July 2026 evaluation table, whose first two rows are input and output price per 1M tokens for all four compared models; per-dollar arithmetic in the section above is computed from those rows, not stated by DeepMind. Machine extraction from a client-side-rendered page, numbers verified across three passes