When Claude Opus 4.7 scores a high 87.6% on SWE-bench Verified, few people ask: what can this score still measure?

On May 5, Cambridge University, ETH Zurich, Stony Brook University, and others jointly released ProgramBench (arXiv:2605.03546), directly targeting the fundamental flaw of existing AI coding benchmarks — they test only bug-fixing, not true from-scratch software construction.

The core logic of SWE-bench is: give the model a GitHub issue and let it fix a specific problem in an existing codebase. This kind of task is essentially targeted debugging — the model knows where the problem is and just needs to find and modify it.

ProgramBench is completely different. It gives the model a compiled executable and English documentation, with no source code, no network access, decompilation forbidden, requiring the model to architect and implement an entire codebase from scratch, making the output behavior consistent with the original program. Tasks cover real large projects from lightweight CLI tools to FFmpeg, SQLite, PHP interpreter — 200 tasks in total.

The evaluation results are sobering: across the 200 tasks, none of the current mainstream models can completely solve any single task. Even Claude Opus 4.7, the best performer, has only a 3% "near-solved" rate (passing 95%+ of behavior tests). This means that on real full-stack software engineering tasks, even the most powerful current models are still far from passing.

Behind this is a worsening industry problem: as models train and overfit on specific benchmarks, the discriminating power of evaluations like SWE-bench Verified is plummeting. In late April, community voices had already declared SWE-bench officially "benchmaxxed" — models learned to recognize test questions rather than truly master skills. Benchmark Hygiene issues became particularly prominent in 2026: the gap between top models on leaderboards is tiny (87.6% vs 80.6%), but these scores don't truly reflect their performance differences in real programming tasks.

ProgramBench's core value isn't scoring models, but revealing a key insight: passing 95% of tests and actually solving an engineering problem are two fundamentally different things. A model good at fixing bugs may not be able to independently complete a full feature development or architecture design.

For AI coding tool developers and users, this means: when selecting tools, don't just look at SWE-bench scores, but combine with stricter end-to-end evaluations like ProgramBench. For teams building AI Coding Agents, ProgramBench may be the real difficulty benchmark worth referencing.

The current 0% full-model solve rate is both a blow and an opportunity — it carves out a truly unconquered territory.