OpenAI retracts its endorsement of SWE-Bench Pro after audit finds roughly 30% of tasks broken

OpenAI has published the results of a quality audit into SWE-Bench Pro, the coding benchmark it previously recommended as a replacement for the widely used SWE-bench Verified, and concluded the eval itself is too flawed to trust. The company is now retracting its earlier endorsement and calling on the community to build better benchmarks.

What's new

OpenAI ran both an automated pipeline and a human review pass over SWE-Bench Pro's task set. The automated pipeline flagged 200 tasks, or 27.4% of the total, as broken; human reviewers went further, identifying 249 tasks — 34.1% of the benchmark — as containing issues. OpenAI grouped the problems into four recurring categories: overly strict tests that fail correct solutions, underspecified prompts that don't give a model enough information to know what's actually being asked, low-coverage tests that let broken code pass anyway, and prompts that are outright misleading about the task.

As OpenAI put it, "An eval should provide meaningful signal through benchmarks that are hard to game, easy to trust, and genuinely reflective of model capability" — a bar the company says SWE-Bench Pro, in its current form, does not clear. OpenAI traces part of the problem to how these benchmarks are built: many of the underlying issues and pull requests were pulled from open-source repositories where they were "originally created for human collaboration, often through long back-and-forths," not written as clean, self-contained evaluation tasks.

Context

SWE-bench Verified has been one of the industry's default yardsticks for agentic coding capability since its release, cited constantly in model launch announcements across labs. As frontier models started saturating it, OpenAI and others pointed to SWE-Bench Pro as a harder, more realistic successor — testing longer-horizon, more complex coding tasks meant to better track real agentic capability rather than memorized patterns. OpenAI's own past commentary had encouraged the field to move toward SWE-Bench Pro for exactly that reason, which makes this retraction a reversal of its own prior guidance, not just a critique of someone else's work.

Why it matters

Coding evals have become a primary marketing surface for frontier labs — a few percentage points on SWE-bench or its successors regularly anchor release announcements and cross-lab comparisons. If close to a third of a widely cited benchmark's tasks are mismeasuring model behavior, then scores derived from it — including any comparisons already published using SWE-Bench Pro — are less reliable than they appeared. OpenAI's decision to publicly retract its own endorsement, rather than quietly stop citing the benchmark, is also notable: it puts pressure on the rest of the industry to audit the evals underpinning their own published numbers instead of treating benchmark scores as settled facts.