[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sci-agent-exam-scientific-agent-benchmark":3,"topics-all":39,"news-related-87d43d7b-b32e-499d-af8a-68afc6506a2c":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":25,"news_slug":32,"published_at":33,"created_at":34,"modified_at":35,"is_published":36,"publish_type":37,"image_url":14,"view_count":38},"87d43d7b-b32e-499d-af8a-68afc6506a2c","科研Agent大考:头名80.9分,达标率仅26%","中科院软件所联合阿里Qwen团队推出科研智能体基准Sci-Agent-Exam:120道专家命题、11个领域、7类任务,程序验证加LLM评审双轨打分。10个前沿模型同考,GPT-6 Astra以80.9分居首,领先第二名13.6分,但运行达标率仅26%,Qwen3.8 Max与Kimi K3进前五。","科研智能体的赛道已经卷到\"替博士生干活\"了,但一直缺一张能服众的考卷:现有基准要么只考答题,要么只考写代码,没人考\"完整的科研工作流\"。10 月 23 日,中科院软件所联合阿里 Qwen 团队、浙大、港科大、清华等机构在 arXiv 发布了 SCI-AGENT-EXAM,给十个前沿模型办了一场 120 道题的科研统考([论文地址](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32441))。\n\n## 考场:156 位专家出题,7 类科研任务\n\n这张考卷搭得相当扎实。156 位领域专家参与命题与验收,出题和评分由互相独立的团队负责;每个任务都由相应领域的博士级专家设计、构建并验证,还会故意注入\"似是而非的干扰项和误导性中间信号\",专门钓那些想走捷径的智能体。\n\n任务覆盖 11 个科学领域——从物理、数学、材料、化学到生物信息、科学计算;再按科研工作流任务分类法(SWTT)切成 7 类:文献综合、假设验证、工具编排、代码生成、错误诊断、端到端分析、结果解读。每个任务要求三份交付物:结构化的最终答案、可执行工件(代码脚本)、文本工件(报告解释),全部在独立 Docker 容器、同一命令行 harness 下受控执行。\n\n打分也是双轨:可执行工件由确定性程序在隐藏数据上重跑验证;文本工件交给专家校准过的 LLM 评审。两路分数分开公布再合并,谁也别想靠一头蒙混。\n\n## 榜单:Astra 大幅领先,国产三家进前十\n\n十个前沿模型同场,分差拉得很开:GPT-6 Astra 综合 80.9 分位居第一,领先第二名 Claude Fable 5 整整 13.6 分;Claude Opus 5 以 64.2 分排第三;Qwen3.8 Max 62.7 分第四、Kimi K3 62.6 分第五;GPT-5.6 Sol 59.6、Claude Opus 4.8 57.1、GLM-5.3 55.9、Grok-4.6 52.4 依次排开,DeepSeek-V4-Pro 50.9 分垫底。\n\n值得注意的是程序分与评审分的错位:Qwen3.8 Max 程序分只有 54.7,LLM 评审却给到 74.2——代码跑分平平,报告写得出彩。这个错位在全榜普遍存在。\n\n## 三个真发现\n\n**达标率悖论。** Astra 均分 80.9 看着漂亮,但按\"单次运行综合分 ≥0.95\"的严格口径,达标率只有 26.0%;第二名 Fable 5 更只剩 9.7%。均值掩盖了方差:今天的科研智能体\"平均能干\",离\"每次都靠谱\"还很远。\n\n**报告强不等于代码对。** 全榜\"报告漂亮但执行拉胯\"(TL)的比例从 0.14 到 0.48 不等;DeepSeek-V4-Pro 以 0.48 的 TL 率和 0.25 的双优率(TR)垫底,是最典型的\"嘴上功夫\"选手;Astra 则以 TR 0.83、TL 0.14 成为最均衡的一个。轨迹分析还发现,多数模型修复\"程序性检查\"比修复\"科学正确性\"更可靠——把结果跑出来容易,把科学道理讲对难。\n\n**贵的反而便宜。** Astra 单次运行成本 20.79 美元拿下最高分,总成本反而低于 Qwen3.8 Max 和 Kimi K3;GLM-5.3 单次 57.96 美元全场最贵,平均要跑 102.4 轮。论文的结论很直接:评智能体的效率要看 run 级总成本,别只盯着 token 单价。\n\n## 偏科生画像\n\n领域维度上,Astra 在 11 个领域里拿下 10 个第一,但其他模型各有绝活也各有死穴:Claude Fable 5 工程能考 80 分、生物信息只有 49;Kimi K3 材料科学 81、物理科学 50。任务类型上,Qwen3.8 Max、GLM-5.3、Kimi K3 在文献综合类任务里表现相对最好——国产模型先在\"读文献\"这一环找到了身位。\n\n## 所以呢\n\n这张考卷本身比名次更有价值:它把\"科研全流程\"拆成 120 个可执行、可验证的任务,而不是又一张答题卡。但也要泼两瓢冷水:一是榜单基于 Claude Code harness 跑出,换一套 harness 结论未必照搬;二是论文附带的代码仓库链接目前还是 404,想复现得等工件放出。当所有厂商都在讲\"AI 科学家\"故事的时候,26% 的达标率是一个提醒:让智能体真正扛住博士级工作流,离及格线都还有距离。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32441","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,19,22],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":18,"color":14},"9112951a-2abb-4214-b63a-385ec7afb2ba","ai-for-science","AI for Science 专题：追踪 AI 在生命科学、化学材料、物理世界模型等科学方向的关键突破",{"id":20,"name":21,"slug":21,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":23,"name":24,"slug":24,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[26],{"id":27,"lang":28,"title":29,"summary":30,"content":31},"393449d7-4a1e-4c20-ae2a-9acd88aa0b07","en","Sci-Agent-Exam: top model scores 80.9, only 26% runs pass","CAS and Alibaba Qwen team release Sci-Agent-Exam: 120 tasks, 11 domains, 7 types. GPT-6 Astra tops at 80.9; only 26% of runs pass the 0.95 bar.","The race for scientific agents has reached the point of \"doing a PhD student's job,\" yet a credible exam has been missing: existing benchmarks test either question answering or coding in isolation, not the full research workflow. On October 23, the Institute of Software at the Chinese Academy of Sciences, together with Alibaba's Qwen team, Zhejiang University, HKUST and Tsinghua, released SCI-AGENT-EXAM on arXiv — a 120-task unified exam for ten frontier models ([paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32441)).\n\n## The Exam: 156 Experts, 7 Research Task Types\n\nThe construction is unusually rigorous. A total of 156 domain experts participated, with task production and evaluation handled by independent groups. Every task was designed, built and verified by a PhD-level expert in the relevant domain, and the authors deliberately injected plausible distractors and misleading intermediate signals — bait for agents looking for shortcuts.\n\nTasks span 11 scientific domains, from physics, mathematics, materials and chemistry to bioinformatics and scientific computation, and are organized by the Scientific Workflow Task Taxonomy (SWTT) into seven types: Literature Synthesis, Hypothesis Validation, Tool Orchestration, Code Generation, Error Diagnosis, End-to-End Analysis, and Result Interpretation. Each task requires three deliverables — a structured final answer, executable artifacts (code and scripts), and textual artifacts (reports and explanations) — all executed under control inside a dedicated Docker container with the same command-line harness.\n\nScoring is dual-track: executable artifacts are re-run and verified by deterministic programs on hidden data, while textual artifacts go to a calibrated, expert-validated LLM judge. The two scores are reported separately before being combined.\n\n## The Leaderboard: Astra Far Ahead, Three Chinese Models in the Top Ten\n\nThe spread is wide. GPT-6 Astra ranks first at 80.9, ahead of second-place Claude Fable 5 by a full 13.6 points; Claude Opus 5 sits third at 64.2; Qwen3.8 Max takes fourth at 62.7 and Kimi K3 fifth at 62.6; GPT-5.6 Sol (59.6), Claude Opus 4.8 (57.1), GLM-5.3 (55.9) and Grok-4.6 (52.4) follow, with DeepSeek-V4-Pro last at 50.9.\n\nNote the mismatch between programmatic and judge scores: Qwen3.8 Max scores only 54.7 on programmatic checks but 74.2 from the LLM judge — mediocre code, sparkling reports. The gap runs through the whole board.\n\n## Three Real Findings\n\n**The pass-rate paradox.** Astra's 80.9 average looks great, but under the strict \"single-run combined score ≥0.95\" bar, only 26.0% of its runs qualify; Fable 5 manages just 9.7%. Averages hide variance: today's scientific agents are \"good on average\" and far from \"reliable every time.\"\n\n**Strong reports ≠ correct code.** The \"strong report but weak execution\" (TL) fraction ranges from 0.14 to 0.48 across models. DeepSeek-V4-Pro posts the highest TL rate (0.48) and the lowest dual-pass rate (TR, 0.25) — the archetypal smooth talker — while Astra is the most balanced with TR 0.83 and TL 0.14. Trajectory analysis adds that most models recover programmatic checks more reliably than scientific correctness: getting a result to run is easier than getting the science right.\n\n**Expensive yet cheaper.** Astra costs $20.79 per run for the top score — a lower total cost than Qwen3.8 Max or Kimi K3. GLM-5.3 is the most expensive at $57.96 per run, averaging 102.4 turns. The paper's conclusion is blunt: evaluate efficiency at the run level, not by token price.\n\n## The Specialist Profiles\n\nAstra leads in 10 of the 11 domains, but others have both strengths and blind spots: Claude Fable 5 scores 80 in Engineering yet 49 in Bioinformatics; Kimi K3 hits 81 in Materials Science but 50 in Physical Science. On task types, Qwen3.8 Max, GLM-5.3 and Kimi K3 do relatively best on Literature Synthesis — Chinese models first find their footing at the \"reading papers\" stage.\n\n## So What\n\nThe exam itself matters more than the ranking: it decomposes the full research workflow into 120 executable, verifiable tasks rather than another answer sheet. Two caveats: the leaderboard was produced on the Claude Code harness, and conclusions may not transfer to other harnesses; and the paper's code repository link still returns 404 at the time of writing, so replication waits for the artifacts. While every vendor tells an \"AI scientist\" story, the 26% pass rate is the reminder: agents that can reliably carry a PhD-level workflow are not yet within reach.","sci-agent-exam-scientific-agent-benchmark","2026-10-26T17:09:05Z","2026-10-26T17:09:36.511533Z","2026-10-26T17:09:36.511547Z",true,"agent",414,[40,48],{"slug":17,"tag_slug":17,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":36,"created_at":46,"modified_at":47},"AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":36,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"158f2181-ecf3-4e5c-ac5a-16e5c45c9d9a","千篇论文建成AI实验室:前沿模型74%胜过人类作者","scientist-world-1000-research-environments","2026-11-08T21:08:04+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"ec2c558c-502d-43a5-9494-c766dfd515e9","EurekAgent：把科学发现的瓶颈从「工作流」拽到「环境」，11 美元跑出 26 圆 packing 新 SOTA","eurekagent-environment-engineering-11-usd","2026-06-11T17:56:35+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"6fd14d23-04ef-43c0-bb9a-fc365a6a6843","OpenAI湿实验基准:榜首66.7%,菌落还是数错","openai-biobasicsbench-wet-lab-agents","2026-11-08T19:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"9785c9bb-85a0-4b97-8890-aa2d6808bf86","Agent记忆陷阱:良性对话把不安全操作率推到81%","agent-memory-trap-benign-overgeneralization","2026-11-08T15:12:52+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"8857d393-a5f7-4b2a-8439-bf87ac324d4f","数学建模基准自曝:12个模型算对却只得49.82%","tb-mathmodel-agentic-verifiable-math-modeling-bench","2026-11-08T03:15:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"ca1e87fb-2e21-4af2-b958-046a7efbefb0","姚期智署名TCSExam:模型考理论CS,跑分跑出新构造","tcsexam-lean-verified-tcs-benchmark","2026-11-06T19:10:36+00:00"]