NVIDIA inference software stack cuts DeepSeek V4 token cost by 5x on Blackwell in one month

NVIDIA published a detailed breakdown on June 30, 2026 of how its layered inference software stack reduced token costs for DeepSeek V4 on Blackwell hardware by up to 5x within one month of deployment, with a combination of techniques delivering up to 20x throughput gains over baseline.

What's new

NVIDIA's inference software operates across three coordinated layers:

• Production operation: distributed serving, orchestration, autoscaling, and memory management
• Application acceleration: high-performance model execution via runtime optimizations including compute/communication overlap and kernel fusion, primarily through TensorRT-LLM
• Infrastructure access: direct exposure of GPU, networking, memory, and system capabilities

The four key optimization techniques and their combined results:

• Disaggregated serving: separates prefill and decode stages for better resource allocation
• Large expert parallelism over NVLink: spreads MoE expert layers across GPUs via NVLink interconnect
• NVFP4 precision: lower-precision inference format optimized for Blackwell
• Multi-token prediction via DFlash speculative decoding: generates multiple tokens per forward pass

Performance figures cited:

• Up to 5x reduction in token costs for DeepSeek V4 on Blackwell in one month
• Up to 20x throughput increase combining all four techniques
• 50% more tokens per second at Baseten using TensorRT-LLM

Named production adopters: Baseten, Cognition, Deep Infra, Together AI, Cursor.

Context

Blackwell began widespread deployment in early 2026. The software-side improvements described are cumulative gains realized within the first month of production-scale Blackwell use. TensorRT-LLM is NVIDIA's open-source inference library; the broader ecosystem (vLLM, SGLang) also runs on CUDA and benefits from the same kernel-level optimization path.

NVIDIA frames the broader shift in infrastructure decision-making: "infrastructure decisions have shifted from peak chip specifications to cost per token: how many useful tokens they can deliver per dollar, per watt and within required latency targets."

Why it matters

The 5x cost reduction for DeepSeek V4 is significant context for the ongoing AI inference cost curve. NVIDIA's core argument is that its competitive advantage is not purely silicon — it is the software stack layered on top, a positioning designed to differentiate against AMD and custom ASIC providers who can match or exceed raw GPU performance but lack an equivalent software ecosystem.

For inference providers and cloud buyers, the 20x throughput claim with combined optimizations would substantially change the economics of running large MoE models at scale. Cursor and Cognition's inclusion as named adopters signals the stack is being tested under real agentic coding workloads, not just benchmark conditions.