NVIDIA's Nemotron 3 Ultra tops open-model agent benchmarks when paired with LangChain's Deep Agents harness

NVIDIA and LangChain say tuning the agent harness around a model — rather than retraining the model itself — pushed Nemotron 3 Ultra to benchmark-leading results among open models on LangChain's agent evaluation suite, at a fraction of the inference cost of top closed models.

What's new

On LangChain's agent eval suite, Nemotron 3 Ultra evaluated with LangChain's Deep Agents harness scored an aggregate 0.86 at a cost of $4.48 per run, versus $43.48 for the next closest-performing model — roughly a 10x cost advantage while reaching accuracy parity with the highest-scoring closed models on business tasks. NVIDIA and LangChain say the gains came from tuning system prompts, tool descriptions, middleware, and how the agent manages context and evaluates intermediate steps, not from further training Nemotron itself. The two companies are packaging the work as NemoClaw, an open reference blueprint for LangChain Deep Agents tuned specifically for Nemotron 3 Ultra, combined with NVIDIA's OpenShell runtime for safely executing agent actions. LangChain says its agent-engineering platform sees more than 200 million monthly downloads.

"The way to build better agents is to keep improving the system around the model," said Harrison Chase, cofounder and CEO of LangChain.

Context

Most of the industry's agent-performance gains over the past two years have come from bigger or better-trained models. NVIDIA and LangChain's blueprint argues for a different lever: harness engineering — the scaffolding of prompts, tool interfaces, and context management around a fixed model — as a comparably powerful way to close the gap with closed frontier systems. It follows a string of recent NVIDIA-LangChain collaborations on agent tooling, including NVIDIA's AI-Q integration for enterprise deep research, and comes as open models increasingly compete on total cost of ownership for agentic workloads rather than raw benchmark scores alone.

Why it matters

A 10x cost reduction at comparable accuracy is a meaningful number for any enterprise running agents at volume, where inference cost scales with every tool call and reasoning step an agent takes. By publishing the tuning approach as an open reference blueprint rather than a closed product, NVIDIA and LangChain are effectively handing other open-model deployments a template for closing the gap with closed APIs through harness design alone — a cheaper lever than waiting for the next model generation, and one that shifts competitive pressure back toward the closed-model providers on cost rather than just capability.