DiffusionGemma: Google open-sources a diffusion-based 26B MoE text model hitting 1,000+ tokens per second on H100

Google has open-sourced DiffusionGemma, an experimental 26-billion-parameter Mixture of Experts text-generation model that ditches token-by-token autoregressive decoding in favor of parallel block generation. Released under an Apache 2.0 license on June 10, DiffusionGemma sustains more than 1,000 tokens per second on a single NVIDIA H100 — roughly four times the throughput of comparable autoregressive models on the same hardware.

What's new

DiffusionGemma is built on the Gemma 4 architecture but replaces standard next-token prediction with text diffusion:

• Architecture: 26B Mixture of Experts, activating only 3.8B parameters per inference forward pass
• Speed: 1,000+ tokens/sec on a single NVIDIA H100; 700+ tokens/sec on an RTX 5090
• Memory: Fits within 18GB VRAM when quantized, enabling consumer GPU deployment
• Parallel generation: Produces 256 tokens per forward pass using bidirectional attention, shifting compute from memory-bandwidth constraints to parallel arithmetic
• License: Apache 2.0
• Frameworks: Hugging Face Transformers, vLLM, MLX; llama.cpp support in progress
• Deployment: Available via Google AI Studio, Google Cloud Gemini Enterprise Agent Platform, and NVIDIA NIM

The speed gains stem from a fundamental change in how text is produced. Standard LLMs generate one token at a time, with each step gated on the previous. DiffusionGemma instead predicts an entire block simultaneously and iteratively refines it — what Google describes as upgrading inference "from a single, sequential typewriter to a massive printing press."

Google is explicit about the trade-off: overall output quality is lower than standard Gemma 4. For applications requiring maximum accuracy, Google recommends sticking with the standard Gemma 4 family. DiffusionGemma is positioned for scenarios where throughput is the binding constraint, not quality ceiling.

Context

Diffusion-based approaches to text generation have been an active research area since 2022, but achieving competitive quality while exceeding autoregressive inference speeds has remained elusive. Most prior work stayed in academic papers or small-scale prototypes. DiffusionGemma is one of the first production-ready open-weight diffusion language models from a major AI lab, built on the Gemma 4 26B MoE backbone released earlier in 2026.

The Gemma family has followed a consistent pattern: Google releases an open-weight baseline under a permissive license, then layers experimental variants — Gemma 2, Gemma 3, PaliGemma, CodeGemma — that push specific capabilities. DiffusionGemma extends this pattern into inference research, treating the open-source release as a signal that diffusion decoding is ready for wider developer evaluation.

On the hardware side, the model's ability to fit in 18GB VRAM at quantized precision means it is deployable on prosumer GPUs like the RTX 5090, lowering the barrier for developers to benchmark it against their own use cases.

Why it matters

DiffusionGemma targets a specific gap: high-throughput streaming scenarios where latency and token cost dominate over output quality. Agentic pipelines that route large volumes of text, draft-and-refine workflows, and real-time streaming interfaces are natural fits.

The larger implication is architectural. Autoregressive decoding has been the default LLM inference paradigm since the original GPT series. If diffusion-based decoding can close the quality gap over successive model generations — as image diffusion models did relative to earlier approaches — the inference economics of language models could shift substantially. A model sustaining 1,000 tokens per second on a single GPU changes what is cost-feasible for latency-sensitive products, and could alter the make-vs-buy calculation for teams currently relying on external APIs to hit throughput targets.