[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-co-design-bench-agent-delivery-gap":3,"topics-all":38,"news-related-9109e6b0-bf85-472d-abf6-ba8e31dc17c5":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"9109e6b0-bf85-472d-abf6-ba8e31dc17c5","Co-Design 基准:七成准则过,26.4% 完整交付","Co-Design Bench 用 273 个真实用户的改应用请求评测十个编码 Agent 端点。论文评测里排第一的 gpt-6-astra 平均通过 69.9% 的验收准则,却只有 26.4% 的任务被完整交付。作者来自复旦大学、阿里巴巴 Token Hub 等机构。","一个基准跑出两个数，中间差着 43.5 个百分点。Co-Design Bench 从 26,314 段真实用户的编程会话里挑出 273 个「改应用」任务来考十个编码 Agent 端点，论文自己跑出来的最好成绩是：逐条验收准则平均过了 69.9%，但真正把任务完整交付下来的只有 26.4%。换句话说，七成以上的任务里至少还留着一两条没做到的要求。[论文原文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.35138)\n\n## 真实请求长什么样\n\n这个基准的题不是编出来的。从 26,314 段真实编程会话里的 39,325 条生产查询出发，去重后剩 37,463 条，再按难度、可验证性和历史失败信号排序，合成 273 个边界清晰的任务。每个任务的起始应用必须能构建、能启动、能渲染，否则直接剪掉。\n\n榜单上排第一的 gpt-6-astra 在多模态编辑赛道拿到 0.754 的严格通过分，完整交付率 41.0%；到了仓库编辑赛道，严格通过分降到 0.667，完整交付率只剩 17.9%。同一套评分口径，换个赛道就掉一半。\n\n任务分两轨：多模态编辑（ME）100 个，带截图标注，附件可以是图像、音视频或文档，共 553 条准则；仓库编辑（RE）173 个，要求跨组件、路由、状态和数据流的协同改动，共 1,778 条准则。合计 2,331 条专家手写的隐藏验收准则，Agent 执行时看不到，基准也不需要标准代码或目标截图。\n\n打分方式是先判后驳：评测员按源码差异、渲染截图和浏览器交互轨迹，给每条准则判 Pass、Partial 或 Fail，再由一个独立的 LLM 验证员专门去挑战「通过」的判定——查有没有 stub、有没有硬编码、入口够不够得着。此外还有两名前端专家人工复核了 200 组任务与模型的输出。\n\n作者还对评分器本身做了压力测试：换三个不同的裁判模型重新打分，绝对分数最多只偏移 0.053，kimi-k3 与 Qwen3.8-27B 在两个赛道上的相对次序在四套评测配置下都没变。裁判是模型，但这个测量看起来是稳的。\n\n## 三件比名次更值得看的事\n\n一是赛道会改排名。两个赛道严格通过分的 Spearman 相关是 0.818，看着挺稳，但 kimi-k3 从 ME 的第四掉到 RE 的第六，Qwen3.8-27B 从第五升到第二。端点之间的均分跨度在 ME 上有 28.7 分，在 RE 上只有 13.0 分，合成一个总榜就把场景差异盖住了。\n\n二是干得多不等于干得好。ME 上 glm-5.2 平均 69 轮、烧掉 7.29M token，拿到 0.572；gpt-6-astra 只用 32 轮、1.24M token，拿到 0.754。RE 上 Qwen3.8-27B 花 57 轮、6.72M token 打出 0.645，还是输给了花 2.33M token 的 gpt-6-astra 的 0.667。论文的说法是，更高的完成度并不需要更大的执行开销。\n\n三是改得越宽越难。涉及一到两类改动的任务严格通过 0.654，三类四类掉到 0.591，零通过率从 0.9% 升到 5.8%。多改几处，漏的就更多。\n\n四是一个反直觉的细节。按附件类型拆开看，文档类任务的严格通过分最高（0.652），可它的完整交付率只有 18.2%，反而低于纯文本任务的 37.0%。准则覆盖率最高的输入模态，恰恰是最难被完整交付的那一种。\n\n## Agent 开始自己装工具链\n\n另一个信号藏在工具调用里。68.5% 的视频任务运行中出现了 OpenCV，31.5% 调了 ffmpeg；音频任务里有 25.0% 调 ffmpeg、4.4% 调 faster-whisper。更直接的是，Agent 会自己去 Hugging Face 上拉预训练感知模型：AST、CLAP、Whisper、Qwen2-Audio、CLIP、DINOv2、OWLv2 都在轨迹里出现过。\n\n前沿模型当大脑，OpenCV 和 Whisper 当眼睛和耳朵——这条路线未必比「更大的原生多模态模型」更远。\n\n顺带一组对照实验：同一批 WebCompass-edit 任务、同一套评分口径，qwen3.7-plus 从单次生成换成 agentic 循环，平均分从 73.6% 跳到 95.1%，全部通过的任务从 49 个涨到 104 个。换执行方式，比换模型见效快。\n\n## 我的判断\n\n榜单只报一个平均分，是最容易让人误判的形式：它把「每条需求都沾一点边」和「真的做完了」混成同一档。这个基准把两个数并排放出来，69.9% 对 26.4%，这道 2.6 倍的裂口比任何名次都更接近真实生产力。\n\n也要说明，上面所有数字都来自论文自己的评测，社区目前没有第三方复现，基准放出的数据集也只有 ME 那 96 个任务。\n\n所以下次再看到一个 Agent 基准只甩一个平均分，先问一句：完全做完的有几个？","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.35138","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":22,"name":23,"slug":23,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"0f2aad11-cc73-4771-8af5-8a87bfb5af96","en","Co-Design Bench: 69.9% criteria passed, 26.4% delivered","Ten coding-agent endpoints, 273 real app-revision tasks. The paper's best passes 69.9% of criteria per task but fully delivers only 26.4% of tasks.","A single benchmark produced two numbers 43.5 percentage points apart. Co-Design Bench drew 273 real \"revise this app\" tasks out of 26,314 actual user coding sessions and ran ten coding-agent endpoints against them. The best score in the authors' own evaluation: 69.9% of acceptance criteria passed on average, but only 26.4% of tasks fully delivered. In other words, more than seven in ten tasks still carried at least one requirement the agent never met. [Full paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.35138)\n\n## What a real request looks like\n\nThe tasks here were not written from scratch. The pipeline starts from 39,325 production queries across 26,314 real coding sessions, deduplicates them down to 37,463, then ranks candidates by difficulty, verifiability and historical failure signals before consolidating them into 273 bounded tasks. Every starting application must build, launch and render, or it gets pruned outright.\n\nThe suite splits into two tracks. Multimodal Editing (ME) holds 100 tasks with screenshot annotations and optional image, audio, video or document attachments, carrying 553 criteria in total. Repository Editing (RE) holds 173 tasks requiring coordinated changes across components, routes, state and data flow, carrying 1,778 criteria. Together that is 2,331 hidden expert-authored acceptance criteria, invisible to the agent during execution. The benchmark needs no gold implementation and no target screenshot.\n\nThe leader in the authors' evaluation, gpt-6-astra, scores 0.754 strict completion on Multimodal Editing with a 41.0% full-delivery rate. On Repository Editing, the same endpoint drops to 0.667 strict and 17.9% full delivery. Identical scoring, different track, roughly half the number.\n\nGrading follows a judge-then-refute protocol. An evaluator grades each criterion Pass, Partial or Fail using source diffs, rendered screenshots and browser interaction traces. Then a separate LLM verifier challenges every positive verdict, hunting for stubs, hardcoded outputs and unreachable entry points. Two frontend experts additionally hand-checked the rubric scores for 200 task-model outputs.\n\nThe authors also stress-tested the grader itself. Rescoring two solvers with three alternative judges shifted absolute graded scores by at most 0.053, and the ME-versus-RE ordering flip between kimi-k3 and Qwen3.8-27B held under all four evaluator configurations. The measurement looks stable even though the judge is an LLM.\n\n## Three things more interesting than the leaderboard\n\nFirst, the track changes the ranking. Strict-completion rankings correlate at Spearman ρ = 0.818 across the two tracks, which looks solid until you look closer: kimi-k3 drops from fourth on ME to sixth on RE, while Qwen3.8-27B climbs from fifth to second. The spread between endpoint means is 28.7 points on ME but only 13.0 on RE, so a single combined leaderboard hides setting-specific strengths.\n\nSecond, doing more work is not the same as doing better. On ME, glm-5.2 averages 69 turns and burns 7.29M tokens for a strict score of 0.572. gpt-6-astra uses 32 turns and 1.24M tokens and reaches 0.754. On RE, Qwen3.8-27B spends 57 turns and 6.72M tokens to score 0.645, still losing to gpt-6-astra's 0.667 at 48 turns and 2.33M tokens. The paper's phrasing is that higher completion does not require uniformly greater execution effort.\n\nThird, the broader the edit, the harder the task. Tasks spanning one or two change types score 0.654 in strict completion; three or four types drop to 0.591. The zero-pass rate rises from 0.9% to 5.8%. Change more places, and more gets missed.\n\nFourth, one counter-intuitive detail. Broken down by attachment modality, document tasks carry the highest strict score at 0.652, yet their full-delivery rate is only 18.2%, below the 37.0% of text-only tasks. The input modality with the best criterion coverage is exactly the hardest one to deliver completely.\n\n## Agents are assembling their own toolchain\n\nAnother signal hides in the tool calls. OpenCV appears in 68.5% of video task runs, and ffmpeg in 31.5%. Among audio runs, 25.0% call ffmpeg and 4.4% call faster-whisper. More directly, agents reach out to Hugging Face and pull down pretrained perception models themselves: AST, CLAP, Whisper, Qwen2-Audio, CLIP, DINOv2 and OWLv2 all show up in the trajectories.\n\nThe frontier model serves as the brain while OpenCV and Whisper become the eyes and ears. That route may be closer than building ever-larger natively multimodal models.\n\nThere is also a controlled comparison worth noting: on the same WebCompass-edit tasks under the same scoring rubric, switching qwen3.7-plus from single-shot generation to an agentic loop lifts mean graded completion from 73.6% to 95.1%, and the count of all-pass tasks from 49 to 104. Changing the execution mode pays off faster than changing the model.\n\n## My read\n\nReporting a single average score is the form most likely to mislead: it collapses \"touched every requirement a little\" and \"actually finished the job\" into the same bucket. This benchmark puts both numbers side by side. 69.9% against 26.4% is a 2.6x gap, and that gap tracks real productivity far better than any ranking does.\n\nOne caveat: every number above comes from the authors' own evaluation, and there is no third-party replication yet. The released dataset covers only the 96 ME tasks.\n\nSo the next time an agent benchmark hands you one average score, ask: how many were actually finished?","co-design-bench-agent-delivery-gap","2026-10-30T05:00:00Z","2026-10-30T05:09:38.088715Z","2026-10-30T05:09:38.088723Z",true,"agent",86,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"278dc1cc-ab13-4de3-9ed4-27f6732dda3c","五个27B组队靠账本编排追平Fable 5,复现只承认打平","gvs5h-ledger-orchestration-qwen-fable-rerun","2026-11-01T09:20:28+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"fd0e593c-bb2c-44f9-8913-16f8dcef6999","苹果8B智能体:真实iOS上追平Gemini-3.1-Pro","apple-online-rl-ui-agent-live-ios","2026-10-31T11:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"db8f2b94-8565-49fe-9fd8-08310f441161","HiSTEVE 1.0:像人一样玩我的世界,只靠屏幕和键鼠","histeve-hierarchical-vla-minecraft-agent","2026-10-26T15:13:44+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"5be712c9-ff84-4e5a-aefa-673f880c8ed1","视频深度研究报告基准:AI排版强内容弱,最高63.3分","omnidr-bench-video-deep-research","2026-10-25T19:15:23+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"205bd813-42eb-4d65-8e8c-5a9af0d8d097","12个模型搜遍真实网页,最高分39%:人类21.1%更低","opends-world-benchmark-deep-search-agents","2026-10-23T11:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"6ca2aad2-fc68-46ab-8120-3eada9c187e2","模型自己操控相机建图:最强 VLM 平均 IoU 只有四成","spatialscout-vlm-active-3d-mapping-benchmark","2026-10-23T03:30:00+00:00"]