Together AI benchmark: Kimi K3 delivers 2.8x more solved tasks per dollar than GPT-5.6 Sol

Together AI ran Moonshot's newly released Kimi K3 head-to-head against OpenAI's GPT-5.6 Sol on 904 real software-engineering rollouts, finding the two models trade wins depending on whether you're optimizing for raw accuracy or cost efficiency.

What's new

Published on Together's blog on July 26, the benchmark used DeepSWE, a software-engineering task suite, to compare the two coding-focused models across single-shot accuracy, multi-attempt accuracy, cost, and reliability.

On a single attempt, GPT-5.6 Sol wins clearly: 72.7% pass@1 versus Kimi K3's 68.5%. But give either model more tries and the picture flips — Kimi K3 overtakes Sol at pass@2 (82.0% vs 81.0%) and pulls further ahead at pass@4 (89.4% vs 85.8%). On cost, Together's numbers show a wide gap: "Kimi K3: $4.65 per rollout. GPT-5.6 Sol: $8.37. Per solved task, Kimi delivers 14.7 solves per $100 versus Sol's 5.3, about 2.8x the work per dollar."

Sol is the more consistent performer, reliably solving 61 tasks across every attempt compared to Kimi K3's 45, while Kimi K3 achieves broader overall coverage. The two models' success/failure patterns correlate weakly (0.46), which Together says makes them good candidates for routing: a cascade that tries Kimi K3 first and escalates to Sol on failure reached roughly 85.6% accuracy — beating either model run alone. By language, Sol leads on Python, TypeScript, and JavaScript, Kimi K3 leads on Rust, and the two tie on Go.

Context

The comparison lands just days after Moonshot AI published Kimi K3's weights on Hugging Face, and while GPT-5.6 Sol is still fresh off its July 9 launch and its subsequent rollout to Amazon Bedrock. Together AI, an inference and fine-tuning platform that hosts both open and proprietary-adjacent models, has an obvious commercial interest in this kind of comparison: model routing across cost/accuracy tiers is a core part of its product pitch, and DeepSWE-style benchmarking data feeds directly into how it tunes routing decisions for customers.

Why it matters

The result is a useful, concrete illustration of a broader shift in the coding-model market: the interesting competitive axis increasingly isn't a single best-model leaderboard position but the price-performance curve across attempt budgets. A model that loses on pass@1 but wins decisively on cost-per-solve and multi-attempt accuracy is a completely different tool depending on whether an engineering team is running one-shot production completions or an agentic loop that can afford retries.

The low correlation between the two models' failure patterns is arguably the more important finding for infrastructure providers: it validates cascading/routing architectures — try the cheap model first, escalate on failure — as a genuine free lunch rather than a marginal optimization, since 85.6% cascade accuracy beats both individual models' ceilings. As open-weight models like Kimi K3 close the quality gap with frontier proprietary models while undercutting them sharply on cost, expect more inference providers to publish comparisons like this one, and more engineering teams to default to routing architectures instead of picking a single model for all coding workloads.