ParallelKernelBench: Frontier LLMs solve under a third of real multi-GPU kernel problems

Researchers at Together AI have released ParallelKernelBench (PKB), a benchmark designed to test whether frontier large language models can generate optimized multi-GPU CUDA kernels. The results from 87 real-world problems are clear: even the best models today solve fewer than a third of them, and only a fraction of those solutions outperform a standard PyTorch baseline.

What's new

ParallelKernelBench draws its 87 problems directly from production machine learning codebases — Megatron-LM, DeepSpeed, and TensorRT-LLM — covering distributed workload patterns including tensor parallelism, context parallelism, and expert parallelism. The benchmark asks models to generate CUDA kernels that are both correct and faster than the standard PyTorch + NCCL implementation.

Key results across frontier models:

"The best frontier model solves 28 of 87 problems, and only 22 of those solutions are faster than the PyTorch + NCCL baseline," the researchers write.

Failure modes differ by model tier. Weaker models produce kernels that fail to compile. Stronger reasoning models more often generate syntactically valid kernels that produce incorrect outputs — a more subtle failure tied to challenges in rank coordination, data partitioning, and collective ordering across GPUs. Generated kernels also tend to rely on copy engines and SM load/store instructions rather than specialized hardware features like TMA and NVLS, limiting achievable performance gains.

Context

Multi-GPU kernel programming is among the most technically demanding tasks in modern ML infrastructure. Writing a correct, performant CUDA kernel for distributed workloads requires deep knowledge of GPU architecture, memory hierarchy, inter-GPU communication protocols, and parallel programming patterns.

ParallelKernelBench is the first benchmark specifically targeting this capability, and its focus on real production codebases — rather than synthetic problems — gives it direct relevance to the teams who build and maintain ML training infrastructure. The benchmark is open source.

Why it matters

If LLMs could reliably generate optimized multi-GPU kernels, the economics of systems-level ML engineering would shift substantially. A smaller team could maintain a wider range of optimized infrastructure without deep specialization. The current results quantify how far that capability remains from production-ready.

There is a notable finding at the tail: models occasionally produced high-performance kernels with no known public optimized reference, including an NVIDIA NeMo-RL GRPO training kernel that outperformed the standard PyTorch implementation. This suggests LLMs may reach genuine performance engineering insights in isolated cases, even if they cannot do so reliably or at scale.

The benchmark lands at a moment when the broader narrative around LLMs and coding has shifted from syntax-level assistance to full system-level generation. ParallelKernelBench provides a concrete capability ceiling for the most technically demanding end of that spectrum.