Stanford wins Databricks' inaugural Grounded Reasoning Cup as agents average under 30% on a fresh Treasury-document benchmark

Databricks has published results from the inaugural Grounded Reasoning Cup, a live competition that tested whether AI agents trained on one enterprise-document benchmark can generalize to an unfamiliar corpus — and found that most could not.

What's new

Eleven academic teams spent two months building agents on Databricks' OfficeQA Pro benchmark, then were tested live, on competition day, against a newly released and previously unseen benchmark: OfficeQA Pro V2, built from roughly 120,000 pages of U.S. Treasury documents. Teams received only 36 hours' notice of the new corpus before being scored.

Stanford University won with 63.3% accuracy, achieved through what Databricks describes as an end-to-end agent optimization strategy combining a library of reusable skills, targeted document-representation fallbacks, and adaptive verification. UMass Amherst placed second and Yale University placed third. Out-of-the-box frontier agents, run without the custom engineering the competing teams applied, averaged less than 30% accuracy on the same test.

Databricks' own reading of the results points away from the underlying model as the deciding factor: "Agent performance depends on the full system, not only the model." The company reports that the average accuracy gap between the top-scoring and lowest-scoring teams using the same underlying model was 30.4 percentage points — a bigger swing than the gap between different models — and attributes it to differences in parsing, retrieval, tool use, verification, and parallelism across each team's system. The competition was run in partnership with USAFacts and the U.S. Treasury Department, with OpenAI, Anthropic, and Google DeepMind named as industry sponsors.

Context

OfficeQA and its harder successor OfficeQA Pro V2 are Databricks' attempt to measure "grounded reasoning": an agent's ability to locate, cross-reference, and reason over specific facts buried in large, messy real-world document sets, rather than reasoning from a short prompt-supplied context. Grounded, document-heavy reasoning is a common enterprise workload — compliance review, financial reporting, government-records lookup — where benchmarks built on clean, curated Q&A pairs tend to overstate real-world agent capability. Held-out evaluations like OfficeQA Pro V2, released just before the competition, are designed specifically to test whether a team's approach generalizes rather than having been tuned to a known test set.

Why it matters

The headline number — sub-30% accuracy for frontier agents run without specialized engineering, on material every team had two months to prepare for a different version of — is a concrete data point against the assumption that stronger frontier models alone are sufficient for reliable enterprise document agents. Databricks' finding that same-model teams varied by more than 30 accuracy points is a sharper claim: it says system engineering around a model can matter more than which model is used at all. For teams building agents against large, evolving document corpora, the result argues against over-indexing on public benchmark scores that don't test generalization to genuinely new material, and toward investing in the retrieval, verification, and tool-use layers that sit around the model.