Currently, the most severe hurdle in operating large language models (LLMs) is not a shortage of processing power in compute units, but rather the data transfer latency stemming from VRAM memory bandwidth. In the autoregressive text generation approach used by typical LLMs, every time a single token is output, the entire set of model parameters must be transferred from VRAM to the processor's compute units.
Under this architecture, the time the processor spends waiting for massive data movement far exceeds the time it actually spends performing computations. Technically speaking, this means the inference process falls into a state that is "memory-bound" rather than "compute-bound." Although the latest consumer GPUs and mobile SoCs are equipped with powerful computational capabilities, memory bandwidth constraints prevent them from fully realizing that potential, resulting in slower inference speeds and degraded responsiveness.
This problem becomes even more pronounced for users running LLMs in local environments, since they cannot access data center-grade GPUs equipped with expensive high-bandwidth memory. The Multi-Token Prediction (MTP) draft models for Gemma 4 that Google has just released represent an attempt to fundamentally solve this structural hardware constraint through software algorithms.
The Mechanism of Speculative Decoding
The core technology underlying MTP is "Speculative Decoding." This method brings about a paradigm shift by completely separating the processes of token "generation" and "verification" in language models. Specifically, inference is performed by combining two components: a "draft model" that has fewer parameters and runs quickly, and a "target model" (the main Gemma 4) with advanced reasoning capabilities.
Once the inference process begins, the lightweight draft model first utilizes surplus computational resources to predict multiple subsequent tokens. This multi-token generation process by the draft model completes in far less time than it would take the target model to process a single token. The resulting candidate token sequence is then passed to the target model, which verifies them in parallel.
If the target model determines that the draft model's predictions are correct, the entire predicted sequence is confirmed in a single forward pass, and the target model itself generates one additional token on top of that. Even if the predictions turn out to be incorrect, the target model immediately corrects them by overwriting with the correct tokens. There is no penalty for incorrect predictions—the draft model's computational results are simply discarded. This mechanism dramatically reduces processor idle time, shifting the balance of processing from memory-bound to compute-bound, and achieving a substantial speedup in inference.
Thorough Optimization from Edge Devices to Workstations
The MTP draft models for Gemma 4 incorporate advanced architectural optimizations designed with real-world deployment environments in mind. One such optimization is the sharing of the KV cache between the draft model and the target model. Because the draft model can seamlessly utilize the target model's activations, there is no need to recompute past context, enabling fast predictions while minimizing the increase in memory usage.
Additionally, for the E2B (2 billion parameter) and E4B (4 billion parameter) models designed to run in resource-constrained environments such as smartphones and IoT devices, final logit computation tends to become a bottleneck. To address this challenge, Google has introduced efficient clustering techniques at the embedder level, further boosting generation speed on edge devices. This translates into practical benefits such as improved responsiveness in mobile applications and reduced battery consumption for on-device AI.
Performance improvements in local development environments and workstations are also significant. Batch size optimization is being advanced not only on high-end GPUs like the NVIDIA A100 but also on unified memory architectures such as Apple silicon (M-series chips). For example, when running a 26B MoE (Mixture-of-Experts) model on Apple silicon, a single request (batch size 1) incurs latency due to expert network routing overhead; however, increasing the batch size from 4 to 8 improves parallel processing efficiency, demonstrating up to a 2.2x speedup in local environments.
The Decisive Advantage of Zero Quality Degradation
An extremely important characteristic of introducing MTP draft models is that the improvement in inference speed comes with absolutely no degradation in output quality. In the AI industry, representative methods for accelerating and lightening models include Quantization, which lowers parameter precision, and Pruning, which removes unnecessary network connections. However, these approaches inevitably involve a trade-off that sacrifices a certain degree of the model's reasoning capability or accuracy.
In contrast, with speculative decoding, the final authority over token verification and output decisions always remains with the main target model. The draft model merely serves as an assistant offering guesses, and in cases where those guesses deviate from context, or where advanced logical reasoning or complex coding is required, the target model generates the correct token itself. This design makes it possible to dramatically reduce inference latency while fully preserving the accurate language comprehension, reasoning ability, and factual fidelity that frontier-class large models possess.
Developer Community Response and Future Implementation Prospects
Google's decision to open-source these MTP draft models under the same Apache 2.0 license as Gemma 4 itself has generated significant buzz in the AI community. Model weights were made publicly available the same day on Hugging Face and Kaggle, and they are also now available for experimentation on Android and iOS platforms via Google AI Edge Gallery.
On developer forums including Reddit, active discussions are taking place regarding the memory usage of this technology. When using MTP, because the draft model must be loaded simultaneously, overall VRAM consumption increases slightly compared to running Gemma 4 alone. However, many engineers have assessed that "the modest additional VRAM consumption is an extremely reasonable trade-off for pushing inference speed up by 2-3x through maximum utilization of computational resources." In fact, the draft model for the E2B model comes in at an extremely compact size of just 78M parameters, which analysts attribute largely to the benefits of Gemma's massive and highly-trained 262k tokenizer.
Support from third-party tools in the open-source ecosystem is also rapidly progressing. Currently, support is being advanced across Hugging Face Transformers, MLX, vLLM, SGLang, and Ollama. Furthermore, development toward MTP support is underway in llama.cpp, which holds tremendous market share and influence as foundational software for local inference, drawing strong anticipation from the community. Once native support in llama.cpp is completed, it is expected to become easy to achieve fast Gemma 4 inference even on consumer GPUs and lower-spec PC environments that lack expensive AI-dedicated hardware, further accelerating the spread of on-device AI.
Open Source Strategy and Significance in the AI Ecosystem
Behind Google's decision to provide the MTP draft models free of charge and to actively promote integration with major open-source tools lies the company's long-term strategy regarding an open ecosystem. As leadership at DeepMind has suggested, beyond contributions to research institutions and academia, there is also a judgment at play that as on-device AI execution becomes mainstream, the business advantages of keeping small models closed are diminishing.
Historically, Google has shaped industry standards and driven the entire ecosystem forward by open-sourcing foundational technologies—Kubernetes, TensorFlow, and the Transformer architecture paper that sparked the current AI boom, to name a few. The release of Gemma 4 and its MTP draft models this time follows in that same lineage. Improvements in open model performance and reductions in inference costs promote autonomous AI development that is not dependent on any specific cloud provider. As a result, a wide range of players, from startups to individual developers, gain access to advanced AI technology, and the pace of innovation in the fields of edge AI and autonomous agent development will continue to accelerate.