Google Gemini Nano acceleration with frozen MTP

On a phone, every token costs battery and every millisecond costs patience. Frozen Multi-Token Prediction speeds up a production model by 50% without retraining it, without a separate drafter, and without changing its output.

Source

Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction

By: Google Research
Published: Jun 26 2026
Type: Engineering article

On-device AI inverts most cloud assumptions. Less memory, less compute, battery pressure, latency pressure, and privacy expectations change what counts as a good engineering trade-off. This article focuses on accelerating models that are already deployed, which is a different problem from training faster models.

How frozen MTP speeds up on-device inference without changing the model

Standard language models generate text one token at a time (autoregressive generation). Each token requires a full forward pass through the model. On a phone, this creates a bottleneck: the processor is underutilised while memory bandwidth is strained, slowing the experience and draining the battery. Features like AI Notification Summaries and Proofread on Pixel phones depend on this being fast.

Speculative decoding is the established approach to fix this. A smaller, faster “drafter” model generates a short sequence of candidate tokens (say, 3 at a time). The main model then verifies these candidates in parallel. If they match, all are accepted in one pass. If not, the system rolls back to the first divergence. The result is more tokens per forward pass of the large model.

The problem with a standalone drafter on a phone is twofold. First, it competes for limited RAM — a 128 million parameter drafter takes memory you don’t have. Second, a standalone drafter is “blind” to the main model’s internal state. It predicts next tokens from text history alone, without the semantic context the main model has already computed.

Multi-Token Prediction (MTP) solves this by replacing the standalone drafter with a lightweight Transformer head attached to the main model’s final layers. Instead of training a separate model, you append MTP heads that take the main model’s final hidden states — the rich activations it has already computed — and use them to predict future tokens autoregressively. The drafter is integrated, not separate.

The “frozen” part is what makes this deployable. Normally, MTP heads are pre-trained alongside the backbone (as with Gemma 4). But you can’t retrain a model that’s already shipped to millions of phones. Google’s approach freezes the Gemini Nano v3 weights entirely and trains only the MTP head parameters to minimise prediction error on future tokens. Because incorrect drafts are always discarded during verification, the final output is bit-for-bit identical to the original model. That means no degradation in capabilities or safety alignment — it’s purely an efficiency optimisation that can ship as an update.

The zero-copy architecture handles the memory constraint. Standard MTP implementations share static parameters like embedding weights, but on-device inference faces a stricter bottleneck: dynamic memory. Even with shared weights, if the drafter maintains its own key-value (KV) cache, it doubles the memory cost. Google engineered the MTP head to cross-attend directly to the main model’s frozen KV cache — no duplicate history. This eliminates drafter prefill latency (no extra time to process the prompt) and saves 130 MB of runtime memory per instance compared to a standalone drafter.

The results on Pixel 9 devices: 50% or more speedup depending on the task, compared to standalone drafters of comparable parameter counts. MTP correctly predicts an average of nearly two additional tokens per inference pass in production workloads. Instruction-following tasks and predictable text structures (like smart replies) see the biggest gains — up to 55% improvement in token acceptance for structured patterns. Fewer verification steps also mean less time waking heavy processors, reducing energy consumption.

The common misunderstanding is that speculative decoding always needs a separate small model. Frozen MTP shows you can retrofit acceleration onto a deployed model by attaching a trained head that reuses the model’s own state — no retraining, no separate drafter, no memory duplication, and no change to the output.

What I want to remember

Remember: Small inference improvements become big product improvements on mobile. Frozen MTP gets 50%+ speedup by attaching a head to an already-deployed model, reusing its KV cache, and producing bit-identical output. If AI features are local, every token, millisecond, and memory allocation matters.