[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-are-bench-research-agent-context-requirements":3,"topics-all":38,"news-related-365a0af2-d564-4eff-bf03-a138cdea0c5f":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"365a0af2-d564-4eff-bf03-a138cdea0c5f","研究智能体大考:只查交付过69%,查全要求剩28%","ARE-bench 从研究者自己的项目切出 75 个任务，编译成 1,173 条二元检查项。七个前沿模型各跑三次，GPT-6.1-Sol 实例成功率 66.22% 居首，Claude Opus 5.5 为 56.89%。只查交付契约三者都约 69%，加入历史与工作区要求后掉到 28.44%–40.00%。","训练跑起来了：loss 非零、参数在更新、曲线还在往下走。按任何常规验收标准，这个智能体都算把活干完了。\n\n论文（arXiv 编号 2611.14364，上海人工智能实验室、复旦大学、华东师范大学、上海交通大学、中国人民大学、北京大学六家机构，11 月 10 日挂出）里那个叫 CylindFormer 的案例，把「跑通」和「修对」的差别摆到了明面上。图像-点云配准训练被修到能继续跑，两次执行都满足「训练恢复、loss 非零、参数在更新」这一组条件，但一条通过了全部 20 项检查，另一条只过 16 项——后者悄悄改了坐标处理、重叠标签和匹配损失，而这些是参考论文里写死的方法要求。论文特意限定了一句：这个对比说的是修复过程中的方法一致性，最终配准精度如何，尚未单独测量。\n\n## 基准长什么样\n\nARE-bench 的题目不是编出来的，是切出来的。作者招募了真实研究者，拿他们自己项目里跑过的智能体会话做骨架：每个任务锚定一句真实用户请求，把之前仍然生效的请求压缩成 Background 段，完整工作区原样回放到那个时刻，不做清理。75 个任务横跨调研、设计、数据与环境、代码、执行、分析、写作七个研究阶段，落到 33 条需求、150 个子需求上（完整分类体系含 207 个子需求），最终编译成 1,173 条二元检查项，平均每个任务 15.6 条。\n\n回放后的工作区平均留 41.1 个文件，65.3% 的任务里带着此前智能体自己写下的文件；36.0 轮旧对话被压成一段状态摘要，其中引用平均 10.3 条仍然有效的早期请求。每个任务平均 2.5 个交付物，指令长 1,677 字符。\n\n## 一把尺子量出两个分数\n\n七个前沿模型，每个任务跑三次，共 225 次运行，判定由 GLM-5.3 裁判模型给出。按实例成功率算（所有适用检查项必须全过），GPT-6.1-Sol 为 66.22%，Claude Opus 5.5 为 56.89%，DeepSeek-V4.1-Flash 39.56%，Qwen3.8-Max 37.78%，GLM-5.3-Flash 30.67%，Kimi-K3 28.00%，Qwen3.8-27B 25.78%。跑在头部的那个，仍有约三分之一的运行没能满足全部要求。\n\n真正扎人的是 Table 8 那组对照：交付物和逐项判定一个不动，只改「哪些检查项算数」。DeepSeek-V4.1-Flash 只查交付契约与通用规范时是 69.33%，加入历史要求和工作区\u002F材料要求后掉到 40.00%，按完整清单算是 39.56%；GLM-5.3-Flash 是 69.33% → 31.56% → 30.67%；Kimi-K3 是 69.78% → 28.44% → 28.00%。DeepSeek 与 Kimi 之间的差距从 0.44 分被拉开到 11.56 分。\n\n## 卡住交付的是正确性\n\n按维度拆开，正确性是七个模型共同的最低项：正确性实例成功率区间 31.11%–71.11%，完整性是 72.00%–87.56%，约束合规是 79.11%–93.78%；85.5%–92.8% 的失败运行里至少含一条没满足的正确性项。DeepSeek-V4.1-Flash 的完整性 CSR 有 93.16%，正确性 ISR 只有 47.11%，整体 39.56%。\n\n阶段之间的落差同样明显：Kimi-K3 代码能力排第三、73.96%，执行和写作却只有 47.47% 和 47.31%，后者在七个模型里垫底。设计与执行则是所有模型都最难啃的两段。\n\n## 更贵不等于更靠谱\n\n按论文的冻结价格，GPT-6.1-Sol 每次运行约 2.17 美元，Claude Opus 5.5 约 5.69 美元，贵 2.62 倍，实例成功率反而低 9.33 个点。便宜的 GLM-5.3-Flash 每次约 0.20 美元拿到 30.67%，DeepSeek-V4.1-Flash 约 0.53 美元拿到 39.56%。上下文长度同样不作数：Claude Opus 5.5 峰值上下文 104.71K、Qwen3.8-27B 是 108.02K，两者接近，实例成功率却差 31.11 个点。\n\n## 作者自己泼的三盆冷水\n\n论文的局限章节写得很实在。跑法不统一：GPT-6.1-Sol 用 Codex harness，其余六个都用 Claude Code，所以跨模型差异反映的是模型与 harness 合起来的效果。评判也不是纯客观：rubric 仍是人工判断，裁判分歧集中在同一个方向、主要是重排相邻名次，独立人工标注被列为下一步。样本也未必代表全体：源会话可能过度集中于某些项目、工具和研究者群体，切点采样不保证覆盖所有研究工作流，回放与依赖恢复还可能丢掉原始研究者当时看得到的上下文；上下文与工作区各自贡献多少，要等后续消融才能拆开。\n\n论文对整份结果也留了限定：这些分数应被读作对这套已发布任务分布的证据，而非普适结论。顺带一提，论文开篇拿 Qwen 团队自报的 Qwen3.8-Max 在 PaperBench Code-Dev 上 93.0% 作反例——那个变体不检查执行，也不检查论文的实证结果有没有被复现。\n\n## 所以呢\n\n这套设计真正的价值不在于又添了一个分数，而在于它把「智能体有没有把项目往前推」和「智能体是不是把项目改跑偏了」拆成了两件可分开算的事。交付物齐了不等于方法没被偷偷换掉——CylindFormer 那两次执行，训练都跑起来了，区别就卡在 20\u002F20 和 16\u002F20 之间。\n\n论文：https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14364 ；项目页：https:\u002F\u002Farebench.com","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14364","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3b053e5f-e7de-4413-95ff-e48f1758484b","en","ARE-bench: Best Research Agent Clears Only 66% of Real Tasks","ARE-bench cuts 75 real researcher tasks into 1,173 checklist items. GPT-6.1-Sol leads at 66.22%; delivery-only checks give 69%, full context 28.44%.","Training resumed. The loss was nonzero, the parameters kept updating, the curve kept moving. By any conventional acceptance check, the agent had finished the job.\n\nA paper (arXiv 2611.14364, from Shanghai Artificial Intelligence Laboratory, Fudan University, East China Normal University, Shanghai Jiao Tong University, Renmin University of China and Peking University, posted on November 10) puts the distance between \"running again\" and \"repaired correctly\" in plain sight. Its CylindFormer case is an image-to-point-cloud registration training job that had to be brought back to life. Two separate executions both satisfy the same operational conditions: training resumes, loss is nonzero, parameters update. One passes all 20 checklist items. The other passes 16. That second run quietly changed coordinate handling, overlap labels and matching losses - all of which the reference paper in the workspace had specified. The authors flag the boundary themselves: the comparison concerns method consistency during recovery, and differences in final registration accuracy still require separate measurement.\n\n## What the benchmark actually is\n\nThe tasks are not written from scratch. They are cut out of real work. The authors recruited actual researchers and used agent sessions from their own projects as the skeleton: every task anchors on one genuine user request, compresses the still-effective earlier requests into a Background section, and replays the complete workspace to that moment without cleaning it up.\n\nThe 75 tasks span seven research stages - Survey, Design, Data & Env., Code, Execution, Analysis and Writing - and map onto 33 requirements and 150 sub-requirements out of a full taxonomy of 207. They compile into 1,173 binary checklist items, 15.6 per task on average.\n\nThe replayed workspace holds 41.1 files on average, and 65.3% of tasks contain files the agent itself wrote in earlier turns. A block of 36.0 earlier user turns is compressed into a state summary that cites 10.3 still-effective earlier requests on average. Each task averages 2.5 deliverables, with instructions 1,677 characters long.\n\n## One ruler, two scores\n\nSeven frontier models, three rollouts per task, 225 task runs in total, with verdicts issued by a GLM-5.3 judge. On instance success rate - every applicable checklist item must pass - GPT-6.1-Sol reaches 66.22%, Claude Opus 5.5 56.89%, DeepSeek-V4.1-Flash 39.56%, Qwen3.8-Max 37.78%, GLM-5.3-Flash 30.67%, Kimi-K3 28.00%, and Qwen3.8-27B 25.78%. Roughly one third of the leader's runs still failed to satisfy every requirement.\n\nThe sharper result is Table 8, where the deliveries and the item-level verdicts are held completely fixed and only the definition of what counts changes. DeepSeek-V4.1-Flash scores 69.33% when only delivery contracts and general norms are checked; adding historical requirements and material-or-workspace requirements drops it to 40.00%, and the full checklist puts it at 39.56%. GLM-5.3-Flash goes 69.33% to 31.56% to 30.67%. Kimi-K3 goes 69.78% to 28.44% to 28.00%. The gap between DeepSeek and Kimi widens from 0.44 points to 11.56 points.\n\n## Correctness is what blocks delivery\n\nBroken out by dimension, correctness is the lowest-scoring dimension for all seven models. Correctness instance success rate ranges from 31.11% to 71.11%, against 72.00%-87.56% for completeness and 79.11%-93.78% for constraint compliance. Between 85.5% and 92.8% of failed runs contain at least one unmet correctness item. DeepSeek-V4.1-Flash reaches 93.16% completeness CSR but only 47.11% correctness ISR, and 39.56% overall.\n\nThe stage spread is just as pronounced: Kimi-K3 ranks third on Code at 73.96%, yet its Execution and Writing scores are 47.47% and 47.31%, the latter last of all seven models. Design and Execution are the hardest stages for every model.\n\n## More expensive is not more reliable\n\nUnder the paper's frozen prices, GPT-6.1-Sol costs about USD 2.17 per run and Claude Opus 5.5 about USD 5.69 - roughly 2.62 times as much for an instance success rate 9.33 points lower. The cheap options do not win either: GLM-5.3-Flash gets 30.67% at roughly USD 0.20 per run, DeepSeek-V4.1-Flash 39.56% at roughly USD 0.53. Context length is no better a predictor: Claude Opus 5.5 peaks at 104.71K and Qwen3.8-27B at 108.02K, close together, yet their instance success rates differ by 31.11 points.\n\n## Three caveats the authors raise themselves\n\nThe limitations section is unusually blunt. The harnesses are not uniform: GPT-6.1-Sol runs on Codex while the other six run on Claude Code, so cross-model differences reflect the model and the harness together. The judging is not purely objective either - the rubric is still written by humans, judge disagreements concentrate in one direction and mostly reorder adjacent models, and independent human annotations are listed as the next step. The sample is not necessarily representative: source sessions may over-represent particular projects, tools and researcher populations, cut-point sampling does not guarantee coverage of all research workflows, and replay plus dependency recovery can lose context that the original researcher could see. Separating the contribution of session context from that of workspace state, the paper says, needs future ablations.\n\nThe authors also bound the whole result: the scores should be read as evidence about this released task distribution rather than as a universal conclusion. Worth noting, the introduction opens with Qwen's own reported 93.0% for Qwen3.8-Max on PaperBench Code-Dev as a counterexample - a variant that does not check execution, and does not check whether the paper's empirical results were reproduced.\n\n## So what\n\nThe real contribution here is not another score. It is the separation of two questions that usually get merged: whether the agent moved the project forward, and whether the agent moved the project off the rails. A complete set of deliverables does not mean the method was not quietly swapped out. In the CylindFormer runs, training came back up in both cases - and the difference lands between 20\u002F20 and 16\u002F20.\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14364 ; project page: https:\u002F\u002Farebench.com","are-bench-research-agent-context-requirements","2026-11-13T01:00:00Z","2026-11-13T01:13:23.583137Z","2026-11-13T01:13:23.583158Z",true,"agent",163,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"6c8ae96a-53e0-4ef7-8609-40b345cdee7c","决策模型实测:自己演砸42.9%,教学生82.9%","decision-model-actor-judge-teacher","2026-11-15T21:45:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"bd350599-7253-4f32-a874-9224bab89a49","多智能体搜索实测：32组对照里委派多数时候反而更差","multi-agent-search-delegation-accuracy","2026-11-15T09:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"a2038f05-8ad2-4b57-bb29-47ff522e09a0","失败轨迹偷走权重:ALFWorld成功率涨7到9.4点","outcome-mass-alignment-multi-turn-agent-rl","2026-11-14T11:20:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"9531b4f7-3f4d-474c-b233-c715925a3e76","8.4B轨迹压过25.3B技能语料:形式化证明中训配方","davinci-prover-agent-native-mid-training","2026-11-14T07:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"343f6cae-ae14-4617-a317-7316748e1d6e","删掉推理链再转发:同档32B模型差51分","deletion-sensitivity-reasoning-trace-routing","2026-11-13T07:05:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"f2723003-43ac-47fe-bb59-49030446919a","28个决策模型大考:多选最高才36.81%","mmdm-decision-model-benchmark","2026-11-12T13:30:00+00:00"]