The Sequence AI of the Week #895: OpenAI's Show Us Where Coding Evals Break
OpenAI's audit of SWE-Bench Pro reveals that approximately 30% of benchmark tasks are defective, questioning the validity of precise scores. The finding leads OpenAI to withdraw its recommendation of the benchmark and underscores the need for more reliable evaluation methods.
OpenAI’s audit of SWE-Bench Pro shows why a precise score can still be a poor measure - and why coding agents may become essential tools for auditing the benchmarks that grade them.
A frontier coding score can look wonderfully precise: 80.3 percent, one decimal place, clean enough to rank models and anchor product claims. But precision is not validity. If a benchmark rejects correct solutions, accepts incomplete ones, or asks for behavior its prompt never specifies, the number is measuring something other than coding ability.
That is the uncomfortable conclusion of OpenAI’s audit of SWE-Bench Pro. The benchmark was built to address weaknesses of earlier coding evaluations: longer-horizon tasks, more realistic repositories, and code intended to reduce training-data contamination. On its 731-task public split, frontier-model performance climbed from 23.3 percent to 80.3 percent in eight months. Instead of treating that curve as unambiguous progress, OpenAI asked a more important question: how much of the result comes from the model, and how much comes from the test?
The answer is striking. OpenAI estimates that roughly 30 percent of the public benchmark is broken. Its agent-assisted audit labeled 200 tasks, or 27.4 percent, as defective. A parallel campaign involving experienced software engineers labeled 249 tasks, or 34.1 percent, as defective. OpenAI has withdrawn its earlier recommendation that the field adopt SWE-Bench Pro.
Coding benchmarks are executable specifications - except when they are not
Read more