Databricks benchmark finds GLM 5.2 statistically tied with Claude Opus 4.8 on real coding tasks

Databricks has published results from an internal benchmark that tests coding agents against real engineering work pulled from its own multi-million-line codebase, rather than synthetic or curated coding puzzles — and the findings complicate the assumption that the priciest frontier model is always the right choice for a coding task.

What's new

Databricks built the benchmark from actual pull requests spanning more than 10 languages, including Python, Go, TypeScript, Scala, Rust, and Java, with roughly a quarter of tasks rated low-complexity and about 60% medium-complexity. To keep the eval honest, engineers manually reviewed every sample, rewrote tests for rigor, and sealed off git history during runs so agents couldn't recover solutions by reading past commits.

Databricks tested a mix of proprietary and open models across multiple agent harnesses. On its tasks, Claude Opus 4.8 scored an 87% success rate at $1.94 per task, while Claude Sonnet 5 scored 81% at $2.09 per task — six points lower despite Sonnet being roughly 1.7x cheaper per token than Opus. As Databricks put it, "Sonnet 5 is ~1.7x cheaper per token than Opus 4.8, but, on our tasks, we found that Sonnet cost $2.09/task vs Opus's $1.94, while scoring six points lower." The open-weight GLM 5.2 landed in the top capability tier at $1.28 per task, which Databricks describes as "statistically tied with Opus 4.8 on quality." The harness used to run a model mattered too: the Pi harness sent roughly 3x less context per turn than alternatives and produced cost differences exceeding 2x for comparable output quality.

Databricks summarized the overall picture with: "The Pareto frontier for coding tasks includes models from OpenAI, Anthropic, and open source...Open models, and GLM 5.2 in particular, are now able to handle even the highest level of task difficulty."

Context

Coding capability has become one of the most closely watched — and most heavily benchmarked — dimensions of frontier model competition, with labs regularly citing SWE-Bench-style scores in launch announcements. But most of those benchmarks are built from public GitHub issues or synthetic tasks, which can be gamed by training-data contamination or simply don't resemble the messy, large-codebase reality most engineering teams work in. Databricks' benchmark is notable for being built entirely from its own private, multi-million-line production codebase, which sidesteps the contamination risk that has increasingly undermined public coding benchmarks.

Why it matters

For any organization deploying coding agents at scale, the practical question isn't "which model benchmarks highest" but "which model-and-harness combination delivers the best result per dollar for a given task." Databricks' data suggests that token pricing is a poor proxy for actual task cost, since a cheaper-per-token model can still cost more per completed task if it needs more turns or retries to reach a correct answer — and that harness engineering (how much context gets sent, how the agent is orchestrated) can move cost and quality as much as the underlying model choice does.

The finding that an open-weight model, GLM 5.2, now sits in the same capability tier as a frontier proprietary model on real production tasks is also a data point in the broader narrative around open models closing the gap with closed frontier labs on practical, revenue-relevant workloads like software engineering — not just on public leaderboards.