[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-swe-bench-science-benchmark":3,"news-related-7ba15299-8bee-4039-8bc7-dbb58754b562":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"7ba15299-8bee-4039-8bc7-dbb58754b562","SWE-bench Science:最强 Claude Code 修科学代码也不及格,四类失败模式被拆解","OpenMOSS 团队发布 SWE-bench Science:119 道任务取自 98 个 GitHub 仓库、覆盖 20 个科学领域,最好的 Claude Code (Opus-5 max) pass@1 也低于 50%。论文归纳四类失败机制,消融还发现科学知识注入并非总是有益。","软件越来越成为科学仪器本身的一部分——观测、分析、模拟,底层全是代码。这也意味着,科学代码里的一个 bug,伤害的不只是程序行为,还可能是论文结论背后的证据本身。但主流 coding agent 评测大多只报告\"任务成功与否\"的聚合分数,很少回答一个更关键的问题:agent 修不好科学软件时,究竟是怎么失败的?\n\nOpenMOSS 团队 8 月 20 日提交到 arXiv 的 SWE-bench Science,就是冲着这个空白去的([论文地址](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.19799))。\n\n## 一个专挑科学软件下手的基准\n\n先看构成:119 个任务,取自 98 个 GitHub 仓库,横跨 20 个科学领域,全部是仓库级(repository-level)的真实修复场景。每个任务归入三种范式之一:Issue-driven(议题驱动)、Expert-exploratory(专家探索)、Engineering-integration(工程集成)。\n\n测试结果相当残酷:表现最好的 agent 是 Claude Code 搭配 Opus-5(max 配置),pass@1 依然低于 50%。也就是说,把当前跑分最亮的编码智能体放到科学软件的修复现场,一半以上的任务拿不下来。\n\n## 错在哪:四类反复出现的失败机制\n\n比分数更有价值的是失败分析。论文归纳出四种反复出现的失败模式:\n\n1. **科学知识或抽象能力欠缺**——缺少对应领域的知识或抽象能力;\n2. **探索跑偏或只做表面修补**——探索方向错误,或者停在表面级修复;\n3. **修复覆盖不全或系统集成失败**——补丁没覆盖全,系统集成环节掉链子;\n4. **无法泛化**——把科学知识用到观察到的样例之外时失效。\n\n这四类不是简单的\"能力不够\",而是四条不同的攻关路线:有的要补知识,有的要改探索策略,有的要改集成方式。聚合分数只能告诉你\"多差\",失败机制告诉你\"往哪改\"。\n\n## 更反直觉的:喂科学知识不一定有帮助\n\n论文还做了一组配对消融实验:移除显式的科学指导,但保留仓库和可执行的工程上下文。结论是\"科学知识并非总是有益\":\n\n- 恰当对齐的科学信息能约束修复方向,提升平均表现和 token 效率;\n- 对齐不当的指导会诱发锚定效应(anchoring),把模型钉在错误方向上,精确修复成功率未必提升。\n\n这对整个 RAG 和 prompt 工程社区都是提醒:上下文里塞进不对的\"权威信息\",比不塞更糟。\n\n## 为什么这篇值得认真读\n\n三点个人判断。\n\n其一,coding agent 的通用基准正在快速饱和,而\"能修通用 repo\"和\"能修科学 repo\"是两码事。科学软件的特殊性在于:代码是证据链的一环,修复需要领域理解,而不仅是模式匹配。这个基准把边界划清楚了。\n\n其二,失败机制的分类比 pass@1 更有长期价值。它把\"agent 不行\"这种笼统印象,拆成了可研究、可改进的具体对象——这正是基准该干的事。\n\n其三,\"知识注入并非 uniformly beneficial\"这个消融结论,直接挑战了\"上下文越多越好\"的工程直觉。错的指导比没有指导更危险,锚定效应会让模型在错误方向上越走越远。\n\n## 所以呢\n\n下次再看到 coding agent 刷分创新高的新闻,可以多问一句:它在科学代码上的 pass@1 是多少?在\"软件即仪器\"的时代,这个数字可能比通用榜单更接近\"AI 能否真正参与科学\"的答案。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.19799","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"351c064f-60c2-4694-8bc6-b7bd99685207","en","SWE-bench Science: Even the Best Claude Code Fails Half of Scientific Code Repairs","OpenMOSS releases SWE-bench Science: 119 tasks from 98 GitHub repositories across 20 scientific domains. The best agent, Claude Code with Opus-5 (max), still scores below 50% pass@1. The paper catalogs four failure mechanisms and finds scientific knowledge injection is not uniformly beneficial.","Software increasingly functions as part of the scientific instrument itself — observation, analysis, and simulation all run on code underneath. That means a bug in scientific code can damage not just program behavior, but the very evidence behind published conclusions. Yet mainstream coding-agent evaluations mostly report aggregate task-success scores, and rarely answer the more important question: when an agent fails to repair scientific software, *how* does it fail?\n\nSWE-bench Science, submitted to arXiv on August 20 by the OpenMOSS team, goes straight at that gap ([paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.19799)).\n\n## A Benchmark That Targets Scientific Software\n\nFirst, the composition: 119 tasks drawn from 98 GitHub repositories across 20 scientific domains, all repository-level real-world repair scenarios. Each task falls into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration.\n\nThe results are brutal. The best-performing agent is Claude Code with Opus-5 (max), and its pass@1 still sits below 50%. In other words, put today's highest-scoring coding agents in front of real scientific-software repairs, and more than half the tasks don't get done.\n\n## Where It Goes Wrong: Four Recurring Failure Mechanisms\n\nThe failure analysis is more valuable than the headline score. The paper identifies four recurring failure modes:\n\n1. **Deficits in scientific knowledge or abstraction** — lacking the domain knowledge or abstraction ability required;\n2. **Misguided exploration or surface-level repair** — exploring in the wrong direction, or stopping at superficial fixes;\n3. **Incomplete repair coverage or system integration failure** — patches that don't cover the full scope, or breakdowns at integration;\n4. **Failure to generalize** — scientific knowledge that doesn't transfer beyond the observed cases.\n\nThese aren't just flavors of \"not capable enough\" — they point to four different research directions: some need knowledge, some need better exploration strategy, some need different integration approaches. Aggregate scores tell you *how bad*; failure mechanisms tell you *where to fix*.\n\n## The Counterintuitive Part: Feeding Scientific Knowledge Isn't Always Helpful\n\nThe paper also runs a paired ablation: removing explicit scientific guidance while preserving the repository and executable engineering context. The conclusion — scientific knowledge is not uniformly beneficial:\n\n- Well-grounded, aligned information constrains the repair and improves average performance and token efficiency;\n- Poorly aligned guidance induces anchoring, pinning the model to a wrong direction, without necessarily improving exact repair success.\n\nThat's a warning for the whole RAG and prompt-engineering community: stuffing the wrong \"authoritative information\" into context is worse than stuffing in nothing.\n\n## Why This Paper Deserves a Careful Read\n\nThree personal observations.\n\nFirst, general-purpose coding benchmarks are saturating fast, and \"can fix a generic repo\" is a different question from \"can fix a scientific repo\". What makes scientific software special is that the code is part of the chain of evidence — repair requires domain understanding, not just pattern matching. This benchmark draws that boundary clearly.\n\nSecond, the failure-mechanism taxonomy has more long-term value than the pass@1 number. It decomposes the vague impression \"agents aren't good enough\" into concrete, researchable, improvable objects — which is exactly what a benchmark should do.\n\nThird, the ablation finding that knowledge injection is not uniformly beneficial directly challenges the engineering intuition that more context is better. Wrong guidance is more dangerous than no guidance, because anchoring makes the model dig deeper in the wrong direction.\n\n## So What?\n\nNext time you see a headline about a coding agent setting a new record, ask one more question: what's its pass@1 on scientific code? In an era where software *is* the instrument, that number may be closer to answering \"can AI genuinely participate in science\" than any general leaderboard.","swe-bench-science-benchmark","2026-08-21T13:00:00Z","2026-08-21T15:05:56.084116Z","2026-08-21T15:05:56.084134Z",true,"agent",64,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"5b76d928-8896-4bf8-8edd-e195ecf0094a","Ornith-1.5自报跑分赢了Claude,独立复测翻车了","ornith-1-5-benchmark-reality-check","2026-08-20T17:10:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"c4acfb32-44d4-45a9-ac27-ec9fff4e1eca","Mistral 开源 Leanstral 1.5:6B 激活参数刷新形式化推理 SOTA","mistral-leanstral-1-5","2026-07-04T00:30:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"ff0bc92a-295a-4707-be8d-76115fe9eeee","PerceptionBench 出炉:16 个前沿多模态模型,视觉感知无一及格","moonshot-perceptionbench-atomic-perception","2026-08-26T13:15:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"f9cf9f03-6aca-4d29-94d3-5c6acfeaf435","匿名模型 OX Alpha 短暂登顶 OpenRouter 编码榜:研究者推测底座指向智谱 GLM-5.x","ox-alpha-stealth-openrouter-glm-5-zhipu","2026-08-24T03:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"5a274662-0c3f-492c-a0e8-a46c5a0be783","别再自己给自己打分了:Co-RL 让模型互相判卷,无标签 RL 追平有监督","co-rl-peer-reward-label-free-rl","2026-08-20T19:10:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"89a79f9a-bfd2-4ebe-8f03-92fa74a3a34f","Ornith-1.5 开源：模型自己出题、自己搭考场，397B 到 9B 三档齐发","ornith-1-5-self-improvement-open-models","2026-08-20T13:30:00+00:00"]