Gemma 4 Multi-Token Prediction Delivers Up to ~3x Faster Token Generation

Gemma 4 can be paired with multi-token prediction drafters that generate several future tokens in parallel using speculative decoding. The target model then verifies the proposed tokens in a single pass, reducing latency. Multi-token prediction drafters are lightweight auxiliary models that address the memory-bandwidth bottleneck during inference. During token generation, processors repeatedly move billions of parameters from VRAM to compute units, which increases latency and underutilizes compute resources, especially on consumer hardware. The approach helps because predicting obvious continuations still costs similar computation as solving complex logic. Google reports improved responsiveness and faster inference across PCs, consumer GPUs, and mobile devices while preserving reasoning quality and accuracy through final verification by the primary model.

"Gemma 4 can be paired with multi-token prediction (MTP) drafters that use speculative decoding to generate multiple tokens in parallel, allowing the model to verify them in a single pass and achieve up to ~3× faster inference without quality loss."

"Multi-token prediction drafters are lightweight auxiliary models that work alongside Gemma 4 to address the LLM memory-bandwidth bottleneck. As Google engineers explain, during inference the processor spends most of its time repeatedly moving billions of parameters from VRAM to compute units for each token. This constant data movement increases latency and leaves compute resources underutilized, particularly on consumer hardware."

"By pairing a heavy target model (e.g., Gemma 4 31B) with a lightweight drafter (the MTP model), we can utilize idle compute to "predict" several future tokens at once with the drafter in less time than it takes for the target model to process just one token. The target model then verifies all of these suggested tokens in parallel."

"Because the primary Gemma 4 model retains the final verification, you get identical frontier-class reasoning and accuracy, just delivered significantly faster."

Read at InfoQ