Datacurve released DeepSWE, a new Coding Agent evaluation framework that audits the popular SWE-Bench Pro benchmark. The audit reveals that 32% of the "correct" answers in SWE-Bench Pro are actually misjudged — the Agent "solved" the task but the evaluation script marked it as wrong, or vice versa.
The "benchmark auditing" highlight: DeepSWE is the first "benchmark auditor" — it runs the SWE-Bench Pro evaluation pipeline with multiple sanity checks, including (1) re-running the test suite to verify the "correct" answer actually passes; (2) checking for "trivial fixes" (e.g., a one-line change that passes the test but doesn't actually fix the bug); (3) checking for "evaluation script bugs" (e.g., a test that fails for the wrong reason). The audit identifies 32% of the "correct" answers as misjudged.
The "32% misjudgment" finding: this is a significant credibility issue for SWE-Bench Pro. The benchmark has been the de facto standard for Coding Agent evaluation, and a 32% misjudgment rate means that the leaderboard rankings are not reliable. Some "worse" models on the leaderboard may actually be better in practice.
The fix: DeepSWE releases an "audited" version of SWE-Bench Pro, with the misjudgments corrected. The audited benchmark is available now, and the authors recommend that Coding Agent vendors use the audited version for their evaluations.
The bigger takeaway: "benchmark auditing" is essential. The "trust the benchmark" assumption is breaking, and the industry needs independent auditing to ensure benchmark integrity. For the industry, this means benchmark developers should invest in audit infrastructure, and benchmark users should prefer audited benchmarks over raw ones.