Encoder-Free Early Fusion

Sources#

Summary#

A multimodal-design choice in Interaction Models: instead of routing audio and video through large, standalone encoders (and audio out through a separate TTS-like decoder), use minimal pre-processing and a single transformer with all components co-trained from scratch. "Encoder-free" is relative — there are light embedding layers — but there's no Whisper-scale audio encoder or separate TTS model.

The components (one 200ms micro-turn)#

Inputs (any subset of text / frame / audio):

Text → token embedding (standard).
Image / video frame → split into 40×40 patches, encoded by an hMLP (Touvron et al. 2022).
Audio → taken in as dMel (Bai et al. 2024), transformed by a light-weight embedding layer ("bag of embeddings").

Single shared Transformer consumes the fused inputs.

Outputs:

Text → unembedding (standard).
Audio → a flow head (Lipman et al. 2022) producing mel.

All components are co-trained from scratch together with the transformer — not stitched from pretrained encoders/decoders.

Why this matters#

Avoids the latency and complexity of large standalone encoders/decoders — important when you have to do this every 200ms (see Time-Aligned Micro-Turns).
Early fusion (everything into one transformer) means the model reasons jointly over modalities rather than over pre-digested encoder outputs — a prerequisite for things like reacting to a visual cue while speaking (see Full-Duplex Interaction).
Co-training from scratch is consistent with The Bitter Lesson: fewer hand-engineered modular boundaries, more learned-end-to-end.

Independent corroboration: Gemma 4 12B (July 2026)#

The design above rested on a single source and a single lab. Gemma 4 (empirical) arrives at the same architecture from an unrelated starting point — and this is the more interesting kind of agreement, because the motivation differs.

TML removed encoders to hit a 200 ms latency budget. DeepMind removed them for memory: "alleviating the need for separate encoders and reducing memory fragmentation" on edge hardware. Same verdict, orthogonal reasons. When two labs optimizing different objectives converge on deleting the same component, the component was probably not load-bearing.

The Gemma 4 12B implementation:

Vision. 48×48×3 RGB patches through a single 35M matmul, replacing a 550M ViT. 2D coordinate-based positional embeddings are added to patch representations before a final LayerNorm.
Audio. The 305M USM-based conformer is entirely discarded. Raw 16 kHz audio is segmented into 40 ms chunks — 640-dimensional vectors — and projected directly into the LLM embedding space. No positional encoding is added at all: audio is already a temporal sequence.

Table 8 is the receipt. The encoder-free 12B reaches 0.063 WER on English FLEURS ASR and 41.9 CorpusBLEU on CoVoST de→en, against the encoder-having E4B's 0.065 and 42.0. Throwing away a 305M conformer cost, on those numbers, nothing. DeepMind's own caption: "competitive audio-text performance can be achieved without a dedicated audio encoder."

Two caveats the report doesn't state#

It is not a controlled ablation. The encoder-free model is the 12B; the encoder-having comparison is the 4.5B-effective E4B. The 12B has more LLM capacity to absorb the work the discarded encoder used to do. The claim "competitive performance is achievable without an encoder" is supported. The stronger claim "removing the encoder is free" is not tested anywhere in the paper — a same-size arm would settle it, and none is run.

There is a measured regression, and it is specific. Cut vision tokens from 1120 (Table 6) to 280 (Table 12) and the 12B degrades out of order with scale:

Model	InfographicVQA Δ	OmniDocBench 1.5 Δ (↓ better)
31B	−9.2	+0.070
26B-A4B	−11.5	+0.120
12B (encoder-free)	−29.7	+0.244
E4B	−15.2	+0.126
E2B	−19.3	+0.206

The 12B falls harder than models both larger and smaller than it — the only ordering violation in the family, and it is the only encoder-free member. The effect is confined to dense-text-in-image tasks: MMMU Pro (−1.4) and MATH-Vision (−3.0) degrade normally. A plausible reading is that a 35M linear projection performs no feature compression, so the model must spend resolution where a ViT would have spent parameters. Reading small text then becomes token-budget-bound.

If that reading is right, encoder-free is not free — it trades encoder parameters for vision tokens, which is an inference-cost trade, not a saving. The report neither reports nor comments on the inversion.

Third instance: Inkling at 975B (July 2026)#

Inkling (vendor-claim) carries the design from a 276B research preview and a 12B edge model to a 975B open-weights foundation model: audio as dMel spectrograms, images as 40×40-pixel patches through a four-layer hMLP, both via lightweight embedding layers "processed jointly with text tokens," trained from scratch on general-domain data — TML explicitly calls the choice "consistent with the interaction model design." Same lab as instance one, so it is scale-transfer rather than fresh corroboration, but it answers a question the first two instances left open: whether encoder-free inputs hold up when the budget allows any architecture. TML reports Inkling among the strongest open-weights audio models (VoiceBench 91.4, MMAU 77.2, AudioMC 56.6 vs 24–38 for the open omni specialists it compares) and strong chart/diagram vision (CharXiv RQ 78.1). Still no matched-size encoder/encoder-free ablation from any of the three instances — the open question below stands.

Counter-instance: Kimi K3 keeps its encoder at 2.8T (July 2026)#

Three instances agreed; the fourth data point disagrees, and it is the largest model of the four. Kimi K3 (vendor-claim) is a 2.8T/104B open MoE with a 401M MoonViT-V2 vision encoder — a dedicated, standalone vision tower, deliberately retained at a scale where nothing forced the choice. It is not an edge model with a memory ceiling and not a latency-bound interaction model; Moonshot had the budget for any architecture and picked the one this page documents labs abandoning.

The comparison that matters is where it wins. The encoder-free 12B's one measured regression was dense text in images — a −29.7 InfographicVQA cliff and the family's only scale-ordering violation — with the proposed explanation that a projection-only path performs no feature compression, so resolution must substitute for parameters. K3 is the strongest model in this corpus on precisely that task family: OmniDocBench 91.1, the top cell in its own 45-model-pair table (ahead of Fable 5's 89.8), and OfficeQA Pro 63.3 on a setup where "each test case provides the agent with the entire PDF corpus, with all PDFs rendered as images and no machine-readable text available." Meanwhile it trails Fable 5 on the perception-and-reasoning vision benchmarks (WorldVQA 51.0 vs 56.7, BabyVision 85.7 vs 90.5, CharXiv 84.8 vs 88.9) — the inverse of the profile you would predict if the encoder were simply general vision capacity.

How much this should move you: not far, and the reason is worth stating rather than resolving. This is not the matched-size ablation the first open question below asks for; it is a different lab, a different scale, a different training corpus, and a vendor's own numbers. Nothing here shows that removing MoonViT-V2 would cost K3 anything, and nothing in the Gemma 4 data shows that adding an encoder would have saved its 12B. What the four instances jointly establish is narrower and more useful than "encoder-free wins": at the ~1T frontier the field has not converged. TML dropped the encoder at 975B and Moonshot kept one at 2.8T, in the same month, both reporting strong vision. Treat "encoder-free is the trend" as a two-lab observation (TML and DeepMind), not a field verdict.

One asymmetry does survive the caveats: every encoder-free instance here also drops the audio encoder, and K3 has no audio path at all (text and image only). The strongest evidence for encoder-free — Gemma 4's Table 8, where discarding a 305M conformer cost 0.002 WER — is audio evidence, and K3 does not contest it.

Connections#

Interaction Models — parent concept
Time-Aligned Micro-Turns — why minimal pre-processing is a hard requirement (200ms budget)
Interaction / Background Model Split — the other half of the architecture
Full-Duplex Interaction — joint multimodal reasoning is what makes visual+audio interjection possible
The Bitter Lesson — "co-train from scratch, drop the modular encoders" is a bitter-lesson move
TML-Interaction-Small — the model that implements this design (dMel audio, 40×40 hMLP patches, flow head), co-trained from scratch
Gemma 4 — the second, independent instance: a 12B open-weight model that discards its 305M audio conformer outright
Inkling — the third instance: the design at 975B open-weight scale, chosen for interaction-model consistency
Inference Efficiency as Capability — memory, not latency, is DeepMind's motivation; and the dense-text regression suggests the saving is really a parameters-for-tokens trade
The Open-Weight Frontier Gap — encoder removal is one of the levers that lets a small dense model compete at all
Kimi (Moonshot AI) — the counter-instance: a 401M MoonViT-V2 encoder retained at 2.8T, leading the corpus on the dense-text-in-image tasks where the encoder-free 12B regressed

Open Questions#

Does an encoder-free model at matched size still match? Neither source runs the ablation. TML co-trains everything from scratch; Gemma 4 freezes encoders on four models and drops them on one, at a different scale. Kimi K3 retains a 401M encoder at 2.8T and leads on dense-text vision, which sharpens the question without answering it — still no matched-size arm anywhere.
Is the dense-text degradation intrinsic to a projection-only vision path, or an artifact of the 12B's particular training run? The prediction is falsifiable: an encoder-free 31B should show the same InfographicVQA cliff at 280 tokens.
TML deletes encoders and co-trains from scratch. Gemma 4 deletes encoders and trains the 12B from scratch, but keeps frozen encoders elsewhere. Which half of "encoder-free + from-scratch" does the work?

Sources#

Interaction Models: A Scalable Approach to Human-AI Collaboration
Gemma 4 Technical Report — §2.3 (encoder-free architecture), Table 8 (audio without an encoder), Tables 6 & 12 (the 280-vs-1120 vision-token regression) (empirical)
Inkling: Our Open-Weights Model — dMel + 4-layer hMLP at 975B, from-scratch general-domain training, audio/vision benchmark tables (vendor-claim)
Kimi K3 Model Card — §2 spec table (MoonViT-V2, 401M, at 2.8T/104B; modality text+image), §3 vision rows (OmniDocBench 91.1, WorldVQA, CharXiv, BabyVision) and footnote 3 (OfficeQA Pro's images-only PDF corpus) (vendor-claim) — the counter-instance