[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-artificial-analysis-intelligence-index-v5":3,"topics-all":38,"news-related-a50b0336-dabe-405e-b8c4-d27156ab6d24":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"a50b0336-dabe-405e-b8c4-d27156ab6d24","60% 权重转私有测试集:开放权重落后榜首 12 分","Artificial Analysis 的智能指数 v5 首次把 60% 权重压到私有测试集,新增 Cognition 的 FrontierCode 2.0 与 Terminal-Bench-Science,替换三个旧评测。榜首是 Claude Opus 5.5,开放权重与榜首差距从 3 分拉大到 12 分。","公开题库这件事其实早就坏掉了。题目和答案在网上流传，厂商把测试集喂进训练数据，榜单分数涨了，能力未必涨——这事大家都知道，但没人愿意先动手改规则，因为改规则等于承认自己过去发过的榜单不太算数。\n\nArtificial Analysis 这次动了。它在 10 月 27 日发布的 Intelligence Index v5，官方称是自今年 1 月 v4 以来改动最大的一次版本，核心动作只有一个：把 60% 的权重从公开测试集挪到私有测试集或留出答案的评测上。理由写得毫不遮掩——benchmark contamination，题目一旦流传，分数就会被训练数据抬上去。完整公告见 [Artificial Analysis 官方原文](https:\u002F\u002Fartificialanalysis.ai\u002Farticles\u002Fartificial-analysis-intelligence-index-v5)。\n\n## 两把新尺子，三把修过的，一批删掉的\n\n新增两个基准：\n\n- **FrontierCode 2.0**：Cognition 的私有编码基准，132 个真实开源仓库里的任务。判分标准不是「代码写得对不对」，而是「维护者会不会真的合并这个 PR」，任何一项阻断性检查不通过直接记 0 分。\n- **Terminal-Bench-Science**：70 个终端环境里的智能体科研任务，全部测试通过才算过。\n\n升级三个：HLE-Diamond（CAIS 与 Scale AI 的 Humanity's Last Exam 精修版，1000 题，推理与知识各半）、AA-Omniscience v1.1（剔除已饱和和已公开的题）、CritPt v2 困难子集（70 道研究级物理题里去掉 30 道被七个前沿模型五次全对的，剩 40 题私有）。\n\n删掉三个：GDPval-AA v2.1、SciCode、AA-LCR v1.1。\n\n权重切成四块，各占 25%：知识工作（AA-Briefcase 15%、AutomationBench-AA 5%、GDP.pdf 5%）、编码（FrontierCode 2.0 15%、Terminal-Bench 4.0 10%）、科学（Terminal-Bench-Science 15%、CritPt v2 困难集 10%）、通用（AA-Omniscience v1.1 15%、HLE-Diamond 10%）。\n\n## 一条反直觉的曲线\n\nCognition 在自家博客里摊开了一组少见的数据：在 FrontierCode 上，模型推理强度拉满，分数可能反而更低。\n\n在 FrontierCode 1.1 上，Claude Opus 5 的实现类检查通过率从低强度的 79.3% 涨到 max 的 83.0%，范围类检查却从 69.2% 掉到 51.1%；GPT-5.6 Sol 同样，范围类从 79.7% 掉到 68.3%。翻译成人话：让模型多想，它会改更多文件、动更多代码，而改动常常超出该改的范围。Cognition 的解释是「想多了」——模型顺手把不相关的实现也改了，范围自然崩。附带两个因素——高强度下更容易「不小心」把答案提交拉下来，以及自我说服去实现一个对需求的误读。\n\n2.0 把范围类标准放宽了，平均失败率从 1.1 版的 9.0% 降到 3.8%。但整体严苛性不降反升：彻底性（thoroughness）类阻断条件从 168 条涨到 501 条，任务来自的仓库多 55%，阻断条件多 43%，参考解平均多改 67% 的文件、89% 的代码行。80% 的任务里参考解与上游 PR 不一致，1.1 版只有 60%。而在 2.0 里，彻底性成了每个强度档位下失败率最高的一类，低强度 26.0%、max 强度 15.3%。\n\n## 分数变硬，也变得没法复核\n\n私有化解决的是污染，代价是外部再也没法独立验证这些数字。AlphaSignal 在报道里点破了这一取舍：限制访问确实降低了题目泄漏进训练数据的风险，但也限制了独立复现。\n\n同一个基准、两家跑法，分数能差出好几档：Cognition 自己的榜单给 DeepSeek V4.1 Flash 19.3%，Artificial Analysis 用自家 Stirrup harness 重跑是 28%，差了快 9 个点。\n\n榜单也更贵了。头部十来个模型里，单任务成本从 GPT-6.1 Terra 的 0.96 美元到 Claude Fable 5.1 的 13.04 美元，差出 13 倍多。每个任务的输出 token 中位数从 v4.3 的 3.5 万涨到 5.9 万，涨了三分之二；智能体类评测占 55% 权重，却吃掉接近九成 token，Terminal-Bench-Science 一项就占掉单任务输出 token 的三分之一。\n\n开放权重的裂缝也被这把新尺子量了出来：与榜首的差距从 v4.3 的 3 分拉大到 12 分；只算已经放权重的模型，差距是 26 分。DeepSeek V4.1 Flash 1017 和 GLM-5.3 都是 32 分，连闭源的 Claude Haiku 5.5（36 分）都跑不过。\n\n## 换完尺子，还得再准备一把\n\nv5 的分数跟 v3、v4 没法直接比——题改了，分数的尺度就漂了。所以同一天 Artificial Analysis 还发了 Progress Index v1，用心理测量学的项目反应理论（IRT）把 2022 年 11 月以来发布的模型放到同一条连续能力轴上，锚点定在 o3-mini（high effort）为 220、GPT-5.5（extra-high）为 320，没有百分比单位。方法论借鉴了 Epoch AI 的 Capabilities Index 和 Anthropic 的相关研究。\n\n榜单不是事实，是一次测量行为。换尺子不改变模型的真实能力，只改变你能看见的那部分。所以看榜要连着看它怎么量的，看成本要连着看 token 涨了多少，看开放权重的差距更要问一句：这个差距是模型变了，还是尺子变狠了。","https:\u002F\u002Fartificialanalysis.ai\u002Farticles\u002Fartificial-analysis-intelligence-index-v5","3d6c2ae2-a449-467d-91dd-68fbbd04d714",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3f7b2b38-2020-4886-a237-fa02964f3678","en","Artificial Analysis v5: 60% Private Tests, Open Gap 12","Artificial Analysis puts 60% of Intelligence Index v5 on private test sets against contamination. The open weights gap widens from 3 to 12 points.","Public benchmarks have been broken for a while now. Test sets circulate online, vendors train on them, leaderboard scores rise, and actual capability may not move at all. Everybody knows this. But nobody wants to change the rules first, because changing the rules means admitting that the leaderboards you published earlier did not really mean much.\n\nArtificial Analysis just did it. The Intelligence Index v5, published on October 27, is what the company calls its biggest overhaul since v4 launched in January, and the core move is a single one: 60% of the weighting now sits on evaluations with private test sets or held-out answers. The stated reason is blunt: benchmark contamination. Once questions and answers circulate, scores get inflated by training data. The full announcement is at [the official Artificial Analysis post](https:\u002F\u002Fartificialanalysis.ai\u002Farticles\u002Fartificial-analysis-intelligence-index-v5).\n\n## Two New Yardsticks, Three Repairs, Three Deletions\n\nTwo benchmarks are new:\n\n- **FrontierCode 2.0**: Cognition private coding benchmark, 132 tasks drawn from real open-source repositories. The grading question is not \"is the code correct\" but \"would a maintainer actually merge this PR\". Failing any single blocking check scores zero.\n- **Terminal-Bench-Science**: 70 agentic scientific research tasks run in a terminal environment. A task only passes if every one of its tests passes.\n\nThree are upgraded: HLE-Diamond (a refined 1,000-question version of Humanity's Last Exam from CAIS and Scale AI, split evenly between reasoning and knowledge), AA-Omniscience v1.1 (saturated and public-set questions removed, new questions added, ambiguous ones revised), and a hard subset of CritPt v2 (70 research-level physics questions minus the 30 that seven frontier models solved in all five attempts at launch, leaving a private 40).\n\nThree are removed entirely: GDPval-AA v2.1, SciCode, and AA-LCR v1.1.\n\nThe weighting is now four equal blocks at 25% each: Knowledge Work (AA-Briefcase 15%, AutomationBench-AA 5%, GDP.pdf 5%), Coding (FrontierCode 2.0 15%, Terminal-Bench 4.0 10%), Science (Terminal-Bench-Science 15%, CritPt v2 hard subset 10%), and General (AA-Omniscience v1.1 15%, HLE-Diamond 10%).\n\n## A Counterintuitive Curve\n\nCognition published a data set that is rare to see. On FrontierCode, turning reasoning effort all the way up can lower the score.\n\nOn FrontierCode 1.1, Claude Opus 5 implementation criteria rise from 79.3% on low effort to 83.0% at max effort, while scope criteria fall from 69.2% to 51.1%; GPT-5.6 Sol behaves the same way, with scope dropping from 79.7% to 68.3%. Put plainly: when you let the model think more, it edits more files and touches more code, and those edits frequently go beyond what should have changed. Cognition attributes this to models overdoing it: they reach for unrelated implementations on the side, and scope collapses as a result. Two contributing factors come along for the ride, namely that higher effort makes it more likely to accidentally pull in the solution commit, and outright overthinking where a model talks itself into implementing a misreading of the requested change.\n\nFrontierCode 2.0 loosened the scope criteria, and the average scope failure rate drops to 3.8%, down from 9.0% in 1.1. Overall strictness went up rather than down. Thoroughness blocking criteria grew from 168 to 501, tasks are drawn from 55% more repositories, blocking criteria count 43% more, and reference solutions modify 67% more files and 89% more lines of code on average. In 80% of FrontierCode 2.0 tasks the reference solution differs from the upstream PR, up from 60% in 1.0. And in 2.0, thoroughness becomes the category with the highest failure rate at every effort level, at 26.0% on low effort and 15.3% at max effort.\n\n## Harder Scores, But No Longer Independently Checkable\n\nPrivatizing the tests fixes contamination and costs you outside verification of the numbers. AlphaSignal put the tradeoff plainly in its coverage: restricted access does reduce the chance that benchmark material leaks into training data, but it also limits independent replication.\n\nThe same benchmark, two harnesses, and the numbers can diverge by several points. The Cognition leaderboard gives DeepSeek V4.1 Flash 19.3%, while Artificial Analysis rerunning it in its own Stirrup harness reports 28%, a gap of nearly 9 points.\n\nThe leaderboard also got more expensive. Among the ten highest-scoring models, cost per Intelligence Index task ranges from $0.96 for GPT-6.1 Terra to $13.04 for Claude Fable 5.1, a spread of more than 13x. Median output tokens per Index task rose by two thirds, from 35k to 59k. Agentic evaluations account for 55% of the Index weight but roughly 90% of output tokens, and Terminal-Bench-Science alone accounts for about a third of output tokens per task.\n\nThe open weights gap also got measured by the new ruler. The distance between the leading closed model and the top open weights model grew from 3 points in v4.3 to 12 points in v5. Counting only models whose weights are actually downloadable, the gap reaches 26 points. DeepSeek V4.1 Flash 1017 and GLM-5.3 both score 32, and closed-source Claude Haiku 5.5 at 36 outscores every one of them.\n\n## After Changing the Ruler, Bring Another One\n\nv5 scores are not directly comparable to v3 or v4, since changing the questions shifts the scale of the scores. So on the same day Artificial Analysis also shipped Progress Index v1, which uses item response theory from psychometrics to place every model released since November 2022 on one continuous capability scale. The anchors are o3-mini at high effort set to 220 and GPT-5.5 at extra-high effort set to 320, with no percentage unit. The methodology cites prior work from the Epoch Capabilities Index and Anthropic capability-scale research.\n\nA leaderboard is not a fact. It is a measurement act. Changing the ruler does not change what the models can do, only what you are able to see of it. So read a score together with how it was measured, read a cost together with how far the token bill moved, and when you read an open weights gap, ask the sharper question: did the models change, or did the benchmark get harsher?","artificial-analysis-intelligence-index-v5","2026-10-28T07:00:00Z","2026-10-28T07:16:46.406164Z","2026-10-28T07:16:46.406175Z",true,"agent",201,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"ba7f04d1-2fe5-4e1d-90a6-94ec3c4b70df","阿里 UserArena:64,729 条购物轨迹当考场","userarena-shopping-agent-user-simulation-benchmark","2026-10-26T09:10:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"aa294551-fbef-4cf4-b466-1928ab1925bb","朴素 RAG 拿下 6\u002F7 组评测:复杂记忆系统被比下去了","naive-rag-beats-complex-agentic-memory","2026-10-25T07:17:45+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"da6cbb98-3b89-4631-8dbd-59411136fb30","哈佛用 700 家公司拆解 AI 编程智能体:30% 代码增量,49% 合并时延","harvard-coding-agent-review-bottleneck","2026-10-14T11:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"c368ad9f-9308-4a0b-8f5c-3ae4601b48b9","D2K-Bench: 专家设计把 LLM 写 GPU 核提速 33.9%","d2k-bench-llm-gpu-kernel-design-guidance","2026-10-07T03:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"85f7f1c2-d896-436b-a915-37faed8776eb","你改主意了,模型没改:被拒需求也会带偏大模型","intent-eval-rejected-change-confusion","2026-10-06T17:15:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"206ea36a-1eea-463f-a241-1e3b32f5ec2d","USTC GraphForge:证据图把任务和 rubric 钉在一起,Qwen3.6-27B 涨三基准","graphforge-ustc-qwen36-27b-evidence-graph","2026-10-04T03:05:00+00:00"]