On July 8, 2026, OpenAI audited SWE-Bench Pro, a leading evaluation benchmark for AI coding agents, and estimated that roughly 30% of the public problems are broken. Some tests fail code that works correctly, while other problems were found to pass incomplete fixes. Just five months earlier, in February, the company had recommended that the industry move from SWE-bench Verified—whose flaws and training data contamination had become severe—to Pro. The fact that this recommendation was withdrawn in less than five months underscores the need for continuous auditing of the grader itself, before comparing model scores.
From a February Recommendation to a July Withdrawal
SWE-Bench Pro builds problems from the change histories of real software repositories, testing whether an AI agent can read a problem statement and fix the code accordingly. According to an explanation Scale AI published in September 2025, of the 1,865 total problems, 731 are public, 858 are private, and 276 are derived from commercial code. The benchmark covers 41 repositories, with each problem involving an average of 107.4 changed lines across 4.1 files. It was designed to measure long-horizon work closer to real-world development, rather than short function-level problems.
When OpenAI recommended discontinuing use of SWE-bench Verified in February 2026, Pro was the successor it pointed to. In an investigation of 138 problems that OpenAI o3 had failed to solve consistently on Verified, 59.4% were found to have serious issues, and there were signs that the model had learned from published problems and their solution code. Pro, by contrast, was expected to be more resistant to contamination since it spans multiple languages and involves longer tasks.
However, the pass rate of state-of-the-art models on Pro's 731 public problems rose from 23.3% to 80.3% over eight months. OpenAI investigated how much of this increase reflected genuine capability improvement versus flaws in the problems themselves. The conclusion was that evaluation results could not be taken at face value as capability gains. The company withdrew its recommendation to adopt Pro and has not put forward a specific successor benchmark.
A Single Character of Whitespace Can Fail a Correct Answer
The flaws found in the audit fall into four categories. The first is overly strict tests that demand implementation details not specified in the problem statement, failing functionally correct fixes. The second is underspecified problem statements, where conditions are embedded only in tests hidden from the model taking the exam. Low-coverage tests were also found—these fail to adequately check the required functionality, letting incomplete code pass. In addition, there were cases where the problem statement steered the model toward behavior opposite to what the tests required.
In one problem drawn from OpenLibrary, the problem statement gave an example showing a single space before the pipe character when outputting a Markdown table of contents entry. However, the hidden test required two spaces. The more faithfully a model followed the instructions, the more likely it was to fail. In this case, what was being measured was not the ability to fix code, but the ability to happen to guess a hidden correct answer.
Errors in the opposite direction are just as troublesome. If test coverage is shallow, an agent that has not fully implemented the required functionality can still earn points. Problems that fail correct fixes underestimate capability, while lenient tests overestimate it. When both types coexist within the same benchmark, the aggregate score reveals neither the direction nor the magnitude of the error. This means the audit's roughly 30% figure cannot be used as a uniform correction factor to apply across all models.
Why 27.4% and 34.1% Don't Match
OpenAI first used an automated filter—examining everything from problem statements to model attempts, grading tests, and failure logs—to flag 286 candidate problems. Next, a Codex-based investigation agent was given the repository and execution environment and conducted repeated audits, including running tests and checking surrounding code. Finally, researchers read the summaries and evidence and judged 200 of the 731 public problems—27.4%—to be broken.
In parallel, five experienced software engineers independently reviewed the same set of flagged candidate problems. They identified 249 problems, or 34.1% of the total, as having issues. Because these two figures do not come from separate populations, they should not be added together. They represent different procedures applied to the same candidate pool, with humans applying a stricter standard.
Even so, the two broadly agree. The defect categories assigned by the investigation agent and the human judgments overlapped 74% of the time. The largest discrepancy was in low-coverage tests, which humans flagged as the primary issue in 9.4% of cases overall, compared to 4.1% for the agent. While the agent can efficiently narrow down large numbers of candidates, humans caught more defects when it came to judging whether a solution genuinely satisfied the specification. This audit should be understood not as an example of AI quality-checking replacing human review, but as an example of AI broadly searching for places that humans then needed to read closely.
Audit the Grader Before Trusting the Leaderboard
The realism and instability of SWE-style benchmarks stem from the same source. GitHub issues, commits, and tests were written for humans developing software together while sharing background knowledge. Extracting problem statements and hidden tests from change histories that include lengthy exchanges can reproduce work close to real practice. But this same process also imports implicit assumptions the original developers took for granted, as well as tests meant to verify a particular implementation rather than the requirement itself, into the exam.
External evaluation organizations have already begun changing their measurement approaches. In June 2026, Artificial Analysis removed SWE-Bench-Pro-Hard-AA from its Coding Agent Index and replaced it with DeepSWE. The current index takes a simple average across 113 DeepSWE problems, 84 Terminal-Bench v2 problems, and 124 SWE-Atlas-QnA problems (which test repository comprehension)—321 problems in total. The organization itself also recommends reading the breakdown alongside the aggregate score, since agents with similar overall scores can have different strengths on different problem types.
For companies choosing coding agents as well, treating public leaderboard rankings as the final word in product selection is risky. Candidate models need to be run under the same conditions—same execution environment, same budget—and re-evaluated against a company's own issues and acceptance tests. And those tests themselves must be checked by humans to ensure they aren't failing valid alternative implementations or passing fixes that satisfy only part of the requirement.
OpenAI has called on the industry to build new benchmarks designed from the outset by experienced developers specifically for capability evaluation. What will make the next benchmark trustworthy is not simply adding harder problems. It depends on whether it can show who independently verified the problem statements and the grader before release, how frequently it is re-audited after release, and how past scores are handled when new flaws are discovered.