Gemma 4 models use a training trick to slash their memory footprint

Following Google’s launch of the laptop-grade Gemma 4 12B model earlier this week, the company is releasing new Gemma 4 model checkpoints with quantization-aware training. Quantization is necessary to reduce the amount of memory required to run lightweight models. The standard method is post-training quantization (PTQ), which quantizes the model after training, but could result in weaker performance. The latest Gemma 4 versions use quantization-aware training (QAT) instead to reduce model quality loss and accelerate decode speed, according to Google’s blog post.

Google says that incorporating quantization into the training process results in checkpoints with better performance than models refined with PTQ. The compressed models run on phones and laptops well thanks to a custom mobile-quantization schema. This involves using pre-calculated settings, 2-bit compression in certain parts of the model, and vocabulary list and short-term memory compression. For the user, this results in a smaller model that consumes less system memory.

There are multiple model sizes available with QAT optimization, include Gemma 4 E2B, Gemma 4 E4B, Gemma 4 12B, Gemma 4 26B A4B, and Gemma 4 31B. The smallest versions, like the text-only Gemma 4 E2B model, require less than a gigabyte of memory to run. These small Gemma 4 checkpoints without intensive resource requirements are ideal for running on phones.

Google shared the approximate memory requirements to load the new Gemma 4 models with QAT in various sizes:

There are four different formats of Gemma 4 QAT models available for download: unquantized QAT checkpoints, GPT-Generated Unified Format (GGUF), mobile-optimized, and Compressed Tensors. These models preserve “similar quality to bfloat16 while dramatically reducing the memory requirements to load the model,” according to Google.

After downloading the Gemma 4 QAT model weights, users can run the checkpoints on their phones, laptops, or desktops. You can find the mobile and desktop models on Hugging Face, as well as in LM Studio.