[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-9382e481-e16b-4925-a0d1-3b24cd8ba22a":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"9382e481-e16b-4925-a0d1-3b24cd8ba22a","OpenAI 复审自家推荐的 SWE-Bench Pro:731 道题里约三成是「坏题」,榜单狂欢该降温","OpenAI 在 7 月 8 日发布了一份罕见的「自我审计」:用 Codex 驱动的 investigator agent 加 5 名资深工程师双盲复核 SWE-Bench Pro 的 731 道公开题。自动管线标记 200 道「坏题」(27.4%),人类评估推到 249 道(34.1%),合并估算约三成题目无法可靠反映模型能力。问题被归为四类:隐藏测试过严、prompt 描述不足、测试覆盖低、题干与测试逻辑相悖,全部出在数据层面,与模型本身无关。OpenAI 明确撤回一年前「推荐社区从 SWE-Bench Verified 切到 SWE-Bench Pro」的立场,这是 12 个月内第二次自我撤回评测建议。更值得警惕的是「数字猛涨」:复审前的八个月里,前沿模型 pass@1 从 23.3% 飙到 80.3%——近三倍的跳跃里,恐怕有一笔要算到「模型越来越懂测题人默认的实现细节」上,而非真实工程能力。把 30% 坏题剔除后再排名,#1 与 #5 的悬殊很可能远没宣传中那么夸张,模型方常用的「又反超 Claude」之类话术也得跟着打折。方法学上更有价值的,是 OpenAI 这套「agent + human」双轨流水线:Codex 已能像审计员一样跑测试、查代码、抓失败模式,把昂贵的人工质检规模化。这预示了未来 LLM 评测的新形态——让 LLM 先帮我们质问基准本身,再由人类挑刺,循环改进。对厂商和开发者最现实的提醒:Coding Agent 在 SWE-Bench Pro 上的亮眼分数,先问一句「它是不是强行往 prompt 暗示的实现细节走」;选型别只盯榜单一两个百分点,看真实仓库任务里的修复率更靠谱。","https:\u002F\u002Fopenai.com\u002Findex\u002Fseparating-signal-from-noise-coding-evaluations\u002F","bd0e0e04-6bcf-4b3e-9a56-62c672308ec9",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"42e59a88-7795-47dc-a334-ef1e72c24347","openai","2026-07-13T00:11:00Z","2026-07-13T00:14:00.641036Z","2026-07-13T00:14:00.641052Z",true,"agent",3]