ClaimArticle

Claude Fable 5 beat GPT-5.6 Sol badly on SWE-Bench Pro (80% vs 64.6%), a result OpenAI downplays by claiming roughly 30% of SWE-bench Pro tasks are broken and urging developers to scrutinize results carefully.

Fable 5 crushed the GPT-5.6 family on SWE-Bench Pro, and OpenAI published a separate audit the day before claiming ~30% of that benchmark's tasks are broken, casting doubt on the result. ✦ AI generated

OpenAI · Simon Willison's Weblog · 2026-07-09 · original ↗

Share card ↓Export .md Export .txt

In light of these results, we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results

Read full article ↗excerpt · fair-use quotation

✳ Claude Fable 5 ✳ GPT-5.6 Sol ✳ SWE-Bench Pro

Related moments

Data◆Article · 36 words

Fable's AI system wrote the first genuine and fastest megakernel ever submitted to KernelBench-Mega, hitting an 18.71X speedup with just one cooperative kernel launch per decoded token, versus rivals that needed 4 to 14 separate launches.

Jack Clark (Import AI) · Import AI

Claim◆Article · 54 words

Coding is only a small part of software engineering, and it's a part that AI won't erase the rest of: through your work you also build confidence, make connections with other people, and develop your personal understanding of the domain, none of which disappear just because coding gets automated.

Kent Beck · The Pragmatic Engineer

Fact◆Article · 36 words

Fable's AI system autonomously wrote a CUDA megakernel that achieves an 18.71x speedup over an optimized PyTorch baseline using a single cooperative kernel launch per token, beating every other frontier model's multi-kernel approach on KernelBench-Mega.

Jack Clark · Import AI