TL;DR
- Google has introduced new assistant models, called “drafters,” that could significantly speed up Gemma 4.
- Drafters work by predicting sections of prompts to the main model, which can focus on processing them in bigger batches.
- This allows the model to use the memory and the compute more efficiently.
Google’s recently launched Gemma 4 edge AI models are especially designed to run locally on consumer-hosted hardware. While favorable from a privacy standpoint, local models can easily hog resources and slow down results, rendering them ineffective. So, Google is now offering a potential solution, which it claims can speed up Gemma 4 models by up to three times.
Google recently released Multi-Token Prediction (MTP) drafters for Gemma 4. These drafters are essentially smaller, assistive models that help the primary model by “predicting” part of the user’s request. These smaller models also work in parallel to the main model to manage the compute more effectively.
Don’t want to miss the best from Android Authority?
How does MTP improve Gemma 4?
The process uses a technique called “Speculative Decoding,” in which the drafter models predict upcoming words in the prompt even before the main Gemma model has read through it. While the drafter moves on to the next sequence of words, the main model verifies the predicted set of words at the same time.
If the model accepts the drafted version, it moves on to verify the next set. If it disagrees, it replaces the incorrect word or chunk.
While the extra work may sound counterintuitive, it’s actually not. Let me give you an oversimplified explanation of why MTP works.
The speed of processing is not just determined by the processing hardware (typically GPU cores) but by the memory bandwidth (VRAM). That’s because the model has to be referenced with each new request. So, by combining multiple words into a single chunk, the model must be referenced only once rather than multiple times, thus, shifting the load from the memory to the processing unit.
In addition to making these changes, Google says it is also working to optimize Gemma 4 models of different weights for specific hardware, such as the Apple Silicon or the popular Nvidia A100.
The MTP drafters for Gemma 4, alongside the primary model, can use platforms such as HuggingFace or Kaggle, tools like Ollama, or through Google’s own AI Edge Gallery on Android or iOS.
Thank you for being part of our community. Read our Comment Policy before posting.
