[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-programbench-200-tasks-zero-solve-claude-opus":3,"topics-all":33,"news-related-a9804bb5-bdb6-42f3-880c-fce2f82dec8c":52},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":20,"news_slug":26,"published_at":27,"created_at":28,"modified_at":29,"is_published":30,"publish_type":31,"image_url":13,"view_count":32},"a9804bb5-bdb6-42f3-880c-fce2f82dec8c","AI编程基准测试面临信任危机：ProgramBench揭示主流评估体系的深层缺陷","当Claude Opus 4.7在SWE-bench Verified上拿下87.6%的高分时，很少有人追问：这个分数还能衡量什么？\n\n5月5日，剑桥大学、苏黎世联邦理工学院、Stony Brook大学等机构联合发布了ProgramBench（arXiv:2605.03546），直指现有AI编程基准的根本性缺陷——它们测的只是修bug，而不是真正的从零构建软件。\n\nSWE-bench的核心逻辑是：给模型一个GitHub issue，让它修复现有代码库中的特定问题。这种任务本质上是定向调试，模型知道问题在哪里，只需要找到并修改它。\n\nProgramBench则完全不同。它给模型一个编译好的可执行文件和英文文档说明，不提供源代码、没有网络访问、禁止反编译，要求模型从零架构并实现整个代码库，使输出行为与原始程序一致。任务涵盖从轻量级CLI工具到FFmpeg、SQLite、PHP解释器等真实大型项目，共200个任务。\n\n评测结果令人警醒：在200个任务上，当前所有主流模型没有任何一个能完全解决任何单一任务。即使是表现最好的Claude Opus 4.7，几乎解决（通过95%以上行为测试）的任务率也仅为3%。这意味着，在真正的全栈软件工程任务上，当前最强大的模型距离及格线依然遥远。\n\n这背后是一个越来越严重的行业问题：随着模型在特定基准上训练和过拟合，SWE-bench Verified等评测的区分度正在急剧下降。4月底，社区已有声音宣布SWE-bench已正式成为benchmaxxed——模型学会了考场上认出题目而非真正掌握技能。Benchmark Hygiene（基准测试卫生）问题在2026年变得尤为突出：排行榜上头部模型差距极小（87.6% vs 80.6%），但这些分数并不能如实反映它们在真实编程任务中的表现差异。\n\nProgramBench的核心价值不是给模型打分，而是揭示了一个关键认知：通过95%测试和解决了这个工程问题，是两件本质不同的事。一个擅长修bug的模型，未必能独立完成一个完整的特性开发或架构设计。\n\n对AI编程工具的开发者和使用者而言，这意味着：选型时不应只看SWE-bench分数，而应结合ProgramBench这类更严苛的端到端评估来看。对于构建AI Coding Agent的团队，ProgramBench或许才是真正有参考价值的难度标杆。\n\n当前0%的全模型解决率，既是打击，也是机会——它划定了一片真正未被征服的领域。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.03546","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17],{"id":11,"name":12,"slug":12,"description":13,"color":13},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[21],{"id":22,"lang":23,"title":24,"summary":25,"content":13},"9c3e15ee-a274-462e-b956-dd92d9a200eb","en","ProgramBench exposes deep flaws in AI coding benchmarks","When Claude Opus 4.7 scores a high 87.6% on SWE-bench Verified, few people ask: what can this score still measure?\n\nOn May 5, Cambridge University, ETH Zurich, Stony Brook University, and others jointly released ProgramBench (arXiv:2605.03546), directly targeting the fundamental flaw of existing AI coding benchmarks — they test only bug-fixing, not true from-scratch software construction.\n\nThe core logic of SWE-bench is: give the model a GitHub issue and let it fix a specific problem in an existing codebase. This kind of task is essentially targeted debugging — the model knows where the problem is and just needs to find and modify it.\n\nProgramBench is completely different. It gives the model a compiled executable and English documentation, with no source code, no network access, decompilation forbidden, requiring the model to architect and implement an entire codebase from scratch, making the output behavior consistent with the original program. Tasks cover real large projects from lightweight CLI tools to FFmpeg, SQLite, PHP interpreter — 200 tasks in total.\n\nThe evaluation results are sobering: across the 200 tasks, none of the current mainstream models can completely solve any single task. Even Claude Opus 4.7, the best performer, has only a 3% \"near-solved\" rate (passing 95%+ of behavior tests). This means that on real full-stack software engineering tasks, even the most powerful current models are still far from passing.\n\nBehind this is a worsening industry problem: as models train and overfit on specific benchmarks, the discriminating power of evaluations like SWE-bench Verified is plummeting. In late April, community voices had already declared SWE-bench officially \"benchmaxxed\" — models learned to recognize test questions rather than truly master skills. Benchmark Hygiene issues became particularly prominent in 2026: the gap between top models on leaderboards is tiny (87.6% vs 80.6%), but these scores don't truly reflect their performance differences in real programming tasks.\n\nProgramBench's core value isn't scoring models, but revealing a key insight: passing 95% of tests and actually solving an engineering problem are two fundamentally different things. A model good at fixing bugs may not be able to independently complete a full feature development or architecture design.\n\nFor AI coding tool developers and users, this means: when selecting tools, don't just look at SWE-bench scores, but combine with stricter end-to-end evaluations like ProgramBench. For teams building AI Coding Agents, ProgramBench may be the real difficulty benchmark worth referencing.\n\nThe current 0% full-model solve rate is both a blow and an opportunity — it carves out a truly unconquered territory.","programbench-200-tasks-zero-solve-claude-opus","2026-05-13T04:01:00Z","2026-05-13T04:08:11.065126Z","2026-08-19T02:08:40.142862Z",true,"agent",284,[34,43],{"slug":35,"tag_slug":35,"title_zh":36,"title_en":37,"intro_zh":38,"intro_en":39,"id":40,"is_active":30,"created_at":41,"modified_at":42},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":44,"tag_slug":44,"title_zh":45,"title_en":46,"intro_zh":47,"intro_en":48,"id":49,"is_active":30,"created_at":50,"modified_at":51},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":53},[54,59,64,69,74,79],{"id":55,"title":56,"news_slug":57,"published_at":58},"da6cbb98-3b89-4631-8dbd-59411136fb30","哈佛用 700 家公司拆解 AI 编程智能体:30% 代码增量,49% 合并时延","harvard-coding-agent-review-bottleneck","2026-10-14T11:30:00+00:00",{"id":60,"title":61,"news_slug":62,"published_at":63},"859242dd-0035-41e6-8db9-4583989048b7","AgentSWE基准:让编码智能体造智能体,无人及格","agentswe-coding-agent-benchmark","2026-10-13T19:08:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"448dbda7-e4ac-44ba-8a40-7af01012b8df","把仓库\"换脸\"再测:SWE-bench 高分最多掉 14.4 分,编码 agent 疑在背题","schrodinger-repo-swe-bench-memorization","2026-09-24T17:11:54+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"781a010e-a14f-415b-ae79-c3e7d6e9a815","RealSWE 基准实测:用户随手一句话,7 个前沿编码模型掉分 6.4 个百分点","realswe-casual-prompts-coding-benchmark","2026-09-06T17:06:25+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"14a7f5ab-e270-461c-b862-4bde139e463f","HarnessDev 基准:让 LLM 自建 Agent Harness,代码领域仍输人类工程师","harnessdev-llm-selfbuilt-agent-harness","2026-09-03T19:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"7ba15299-8bee-4039-8bc7-dbb58754b562","SWE-bench Science:最强 Claude Code 修科学代码也不及格,四类失败模式被拆解","swe-bench-science-benchmark","2026-08-21T13:00:00+00:00"]