Google has launched experimental Multi-Token Prediction (MTP) drafters for the Gemma 4 open-source model family, achieving up to 3× inference speedup on consumer-grade GPUs. This is the most important technical iteration since Gemma 4's spring release this year, and another attempt by Google to bring flagship-model optimization technology down to local inference scenarios.

The essence of speculative decoding: small model "drafts" for the large model

Traditional large language models generate token by token, with the same compute amount per token — whether it's a filler word or a key reasoning step. This means that when running tens-of-B-parameter models on consumer-grade GPUs, VRAM bandwidth often becomes the bottleneck: the time the processor waits for weight data to move from VRAM to the compute unit far exceeds the actual compute time.

The MTP drafters' approach is clever: a small model (only 74 million parameters) "guesses" multiple subsequent tokens for the large model in advance, then the main model verifies these guesses in parallel. If correct, skip the latter's computation; if wrong, the main model takes over the correct path. Gemma 4's E2B and E4B drafters also share KV Cache — the main model's already-computed context doesn't need recomputation, plus sparse decoding technology compresses the token candidate range to high-probability clusters, further reducing compute overhead.

Real-world data: 3× speedup on RTX PRO 6000

Ars Technica reports comparison testing on NVIDIA RTX PRO 6000 graphics cards: standard autoregressive inference vs Gemma 4 26B with MTP Drafter enabled, same output quality, half the wait time, translating to about 3× more tokens generated per second. This number is consistent with the speculative-decoding speedup data Google TPU team previously published, but this time it's targeting local GPU deployment scenarios.

Open source and efficiency, Google's two legs

Another noteworthy aspect of this Gemma 4 update is its positioning: Google simultaneously switched the license to Apache 2.0, fully friendly to commercial use. Against the backdrop of increasingly fierce competition from open-source models like Llama and Mistral, inference efficiency optimization has become a key differentiation battlefield — models not only need to run, but run fast. The emergence of MTP drafters shows Google is systematically porting Gemini series frontier optimization technology to open-source Gemma models.

This also echoes one of the main themes of the 2026 LLM industry overall: a shift from "the bigger the model the better" to "the cheaper the inference the better." Whether speculative decoding, KV Cache compression, or quantization, the core contradiction is the same — how to squeeze more effective compute out of consumer-grade hardware. Gemma 4's MTP drafters is the latest, most concrete case in this trend.

For local-AI players, Gemma 4 MTP drafters is open on Hugging Face; the 74M-parameter small size means even mid-range devices beyond Raspberry Pi can benefit. If you're running Gemma 4 locally, consider updating to the latest version to try — the speedup may exceed expectations.