Perplexity releases Q2D-Web, a 190-million-document benchmark for agentic web retrieval

Perplexity introduced Q2D-Web on September 9, 2026, a large-scale benchmark and public leaderboard for evaluating how well embedding models retrieve documents inside agentic retrieval-augmented generation (RAG) systems, built from real production search traffic rather than synthetic queries.

What's new

Perplexity's own description: "Today, we are introducing Q2D-Web (Query2Doc-Web), a private benchmark and public leaderboard for evaluating retrieval in agentic RAG systems." On scale, the company states: "Q2D-Web is built to evaluate embedding models on large-scale web search. It consists of 190 million web documents and 69,721 agent-reformulated queries in ten languages, sampled over nine months of PII-free production search traffic."

The benchmark ships with three separate sets of relevance judgments: raw agent citations, production ranking signals, and a combined set that unions both and adds LLM-based judgments of unlabeled documents to cut down on false negatives. Queries span ten languages, with English making up the largest share, followed by Spanish, Russian, German, French, Portuguese, Italian, Korean, Japanese, and Chinese. On the initial leaderboard, Perplexity's own pplx-embed-v1-4b leads the Web Ranking and Combined categories, while NVIDIA's Nemotron-3-Embed-8B leads on citation relevance. The leaderboard is hosted on Hugging Face, and outside teams can submit their own models for evaluation.

Context

Q2D-Web extends Perplexity's earlier Q2D benchmark to a much larger scale and ties it directly to how the company's own product actually gets used: the queries are drawn from real agent-reformulated searches rather than hand-written test sets, and the relevance judgments incorporate what real users and agents actually clicked on or cited. Retrieval benchmarks built from held-out academic corpora have historically been criticized for not reflecting how retrieval systems behave once agents start rewriting and chaining queries on their own, which is the gap this release is aimed at closing.

Why it matters

As more products route queries through an agent that reformulates and re-issues them before a document is ever retrieved, benchmarks built on single, human-written queries increasingly understate how retrieval actually fails in production. A 190-million-document, production-traffic benchmark gives the broader embedding-model research community a shared, harder target to test against, and Perplexity publishing its own model's results alongside competitors' on the same leaderboard is a visible bet that pplx-embed-v1-4b holds up under scrutiny rather than only performing well on benchmarks the company controls end to end.