[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-system1-arena-decision-model-benchmark":3,"topics-all":38,"news-related-b809f974-e141-4509-96c9-f71e8570a4a0":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"b809f974-e141-4509-96c9-f71e8570a4a0","System1-Arena:30 模型同场比校准","UT Austin 10 月 15 日放出 System1-Arena,30 个模型同场跑 92k 题、105 任务、138 闭环环境。静态决策上决策模型精度逼近 GPT-6 Luna、校准更好;闭环任务 Gemma-4-31B 32.9% 成功率,小决策模型最多 3.1%。","Jev、Microsoft-Decision-1、OpenAI Decisions API 这一波决策模型自 9 月以来密集发布,行业里突然多出一个新问题:它们和 LLM 比到底差多少?JevBench 和 ReflexBench 各自只有几百道题,样本小得让排名随手挑几个任务就能翻盘。System1-Arena 想给这件事一个统一答案。\n\n## 92k 题 + 138 闭环,史上第一个真公平评测\n\nSystem1-Arena 把每个任务定义成一个状态加若干类型化问题的「决策请求」,问题是布尔题、多选(2-255 个选项)或者有序评分(2-10 个等级),模型只返回每个问题的概率分布,不生成文字。评测集包含 92,125 道题目,来自 268 个公开数据源、覆盖 16 个领域的 105 项任务;再加上 138 个来自 18 个家族的闭环环境,包括网络购物、SWE-bench 代码控制、机器人、游戏和驾驶,其中实时模式下环境会在模型推理过程中继续演化。30 个模型被统一打分——7 个闭源 LLM、13 个开源 LLM 通过答案 token 取概率、10 个 System-1 决策模型直接接决策接口。\n\n## 静态决策:大决策模型和快闭源 LLM 几乎打平\n\n非 Agent 类任务上,Decision-2.0-Vega-27B(Jev 系)和 Jev 分别拿到 73.0% 和 72.6% 的总体精度,跑赢 GPT-6 Luna 的 71.2% 和 GPT-5.6 Luna 的 70.0%。两者距离开源最强的 Gemma-4-31B(73.9%)只剩 0.9 到 1.3 个百分点。Brier 分数上,这两个决策模型拿到 0.343 和 0.349,只有四个比它们更准的模型全部是闭源 LLM,且每个的成本是 GPT-6 Luna 的 20 到 105 倍。Decision-2.0-Vega-27B 每千次请求约 0.05 美元,Microsoft-Decision-1 低至 0.038 美元,而精度最高(81.1%)的 Claude Opus 5.5 每千次请求要 10.38 美元。\n\n## 思考能提精度,但会破坏概率校准\n\n论文给四个开源 LLM 设了 0 到 2048 个推理 token 的预算。零思考下,它们都比 Jev 差(59.3% 到 70.2% vs Jev 72.6%)。小预算反而有时会拉低精度——DiffusionGemma 在 128 个 token 时直接从 69.7% 掉到 56.3%。把预算拉满到 2048,三个模型超过 Jev,Qwen3.8-27B 从 70.2% 升到 75.6%。但同一时间,Qwen3.8-27B 的 Brier 分数从 0.391 退化到 0.430,平均置信度从 81.4% 飙到 97.4%。模型把越来越多的概率质量压到单一选项上,即使错的时候也错得自信,Brier 罚得更狠。换句话说,想拿 reasoning 提精度,得接受概率不再可信。\n\n## 闭环任务:大模型赢面大,小决策模型基本不上桌\n\n闭环任务完全改了榜单。九个模型在单步决策上精度挤在 67.4% 到 71.3% 之间,但闭环成功率从 5.7% 一路拉到 32.9%。Decision-2.0-Vega-27B、Jev、Qwen3.5-9B 三者单步都是 67.8%,闭环成功率却分别是 26.5%、17.4% 和 5.7%。错误会在交互里累积,静态正确不代表长链路能走通。开源大模型 Gemma-4-31B 拿下闭环第一名 32.9%,四个小于 10B 参数的开源 LLM 全部低于 6%。Jev 闭环 17.4%,好于六个开源 LLM,但被两个 Luna 模型(22.9% 和 18.9%)压在身后。\n\n## 这件事意味着什么\n\n决策模型不是「更便宜的 LLM」,也不是「更小更省」的同义词。它们是一类专门为结构化概率输出而生的模型,优势在「短决策 + 低延迟 + 可校准」,劣势在「长链路 + 多步规划」。System1-Arena 把这件事量化得很清楚:在前端路由、内容审核、客服工单分流这类需要稳定置信度的场景,Decision-2.0-Vega-27B、Jev、Microsoft-Decision-1 已经够用,而且校准比 GPT-6 Luna 还稳;但放到 Agent 长链路里,临时抽取系统 1 之后还得回头让系统 2 接手。\n\n这其实印证了 Kahneman 四十年前在《思考,快与慢》里写下的框架:系统 1 处理熟悉的、可重复的判断,系统 2 处理陌生的、需要规划的判断。System1-Arena 让这条经验法则第一次有了 30 个模型、9 万多道题的数据支撑。Agentic AI 真正要解决的,不是「让模型替你判断」,而是「让模型替你判断哪些该交给系统 1,哪些必须留给系统 2」。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.21244","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"08c2f68a-acb6-4479-bdcf-82729340b275","en","System1-Arena: First Benchmark for 30 Decision Models","System1-Arena: 30 AI models on 92k questions, 138 closed loops. Decision models match GPT-6 Luna, calibrate better; loop success drops 32.9% to 3.1%.","Jev, Microsoft-Decision-1, and the OpenAI Decisions API have been released at a pace since September that the industry is now asking a sharp question: how do these decision models actually compare to general LLMs? Existing benchmarks (JevBench and ReflexBench) each contain only a few hundred questions, so rankings flip whenever a handful of tasks are swapped. System1-Arena is built to give the field one shared scoreboard.\n\n## 92k questions plus 138 closed loops, the first truly fair benchmark\n\nSystem1-Arena represents every task as a state plus typed questions: boolean, multiple choice (2 to 255 options), or ordered scores (2 to 10 levels). A model returns a probability distribution per question, no text generation. The static portion alone contains 92,125 questions across 105 tasks from 268 public sources in 16 domains. The agentic portion adds 138 closed-loop environments from 18 families, covering web shopping, SWE-bench coding control, robotics, games, and driving. In real-time mode, the world keeps evolving during the model call. Thirty models share the same evaluation: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 decision models that consume the typed interface directly.\n\n## Static decisions: large decision models nearly match fast closed LLMs\n\nOn non-agentic tasks, Decision-2.0-Vega-27B (Jev family) and Jev reach 73.0% and 72.6% overall accuracy, beating GPT-6 Luna at 71.2% and GPT-5.6 Luna at 70.0%. Both sit within 0.9 to 1.3 points of Gemma-4-31B (73.9%), the strongest open LLM. On Brier score, the two decision models post 0.343 and 0.349. Only four models do better, all closed LLMs, each costing 20 to 105 times GPT-6 Luna. Decision-2.0-Vega-27B runs about 0.05 USD per 1,000 requests; Microsoft-Decision-1 drops to 0.038 USD. The most accurate model, Claude Opus 5.5 at 81.1%, costs 10.38 USD per 1,000 requests.\n\n## Reasoning lifts accuracy but breaks probability calibration\n\nThe paper gives four open LLMs a budget from 0 to 2,048 reasoning tokens. With zero thinking tokens, all four underperform Jev (59.3% to 70.2% versus Jev 72.6%). Small budgets sometimes make things worse: DiffusionGemma falls from 69.7% to 56.3% at 128 tokens. At 2,048 tokens, three of the four surpass Jev; Qwen3.8-27B climbs from 70.2% to 75.6%. The same model, however, sees its Brier score worsen from 0.391 to 0.430, and mean confidence jumps from 81.4% to 97.4%. The model concentrates probability mass on a single option, including when that option is wrong, and the Brier penalty grows. The trade is real: reasoning can buy accuracy, but it spends calibration.\n\n## Closed loops: large models still win, small decision models are out of their depth\n\nThe closed-loop leaderboard looks completely different. Nine models cluster between 67.4% and 71.3% step accuracy, yet their closed-loop success ranges from 5.7% to 32.9%. Decision-2.0-Vega-27B, Jev, and Qwen3.5-9B all post 67.8% step accuracy, but their loop success is 26.5%, 17.4%, and 5.7%, respectively. Errors compound over an episode, and a static step does not predict a closed-loop rollout. Gemma-4-31B leads closed loops at 32.9%. The four open LLMs below 10B parameters all sit under 6%. Jev at 17.4% beats six of the open LLMs but trails both Luna models (22.9% and 18.9%).\n\n## What this means\n\nDecision models are not \"cheaper LLMs\" and not \"smaller and thriftier\". They are a distinct class built for structured probability output, with strengths in short decisions, low latency, and calibrated confidence, and weaknesses in long-horizon planning. System1-Arena makes the boundary quantitative: for routing, content moderation, and ticket triage where stable confidence matters, Decision-2.0-Vega-27B, Jev, and Microsoft-Decision-1 are already enough, and their calibration beats GPT-6 Luna. For agent loops, calling on System 1 is only a temporary shortcut before System 2 has to take back over.\n\nThis matches the framework Kahneman laid out in Thinking, Fast and Slow forty years ago: System 1 handles familiar, repeatable judgments; System 2 handles unfamiliar ones that need planning. System1-Arena gives that intuition its first 30-model, 90,000-question dataset. The real question for agentic AI is not \"let the model decide for you\" but \"let the model decide what should be left to System 1 and what must stay with System 2\".","system1-arena-decision-model-benchmark","2026-10-16T03:00:00Z","2026-10-16T03:10:31.444686Z","2026-10-16T03:10:31.444697Z",true,"agent",232,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"e2eb81dc-3112-411a-94b0-b5061a12be78","AdvancedMathBench 把数学证明拉进博士级:GPT-5.5-xhigh 仍只 75.8","advanced-math-bench-phd-level","2026-07-14T16:15:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"fc1e888d-4bee-4633-86f4-edc76aa48161","BlockSearch 把语言模型变成「上下文检索器」：0.6B 在百万 token 上打平向量检索","blocksearch-context-retriever","2026-07-03T06:25:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"95b04c15-d5e5-4dca-ab2b-14e343bdd4e6","UC Berkeley 曝光 AI 基准测试系统性漏洞：45 种方法可在 13 个主流榜单上「不解决任何问题拿满分」","uc-berkeley-benchmark-45-cheats-13-leaderboards","2026-05-15T01:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"9c2b49ce-a543-48cb-932d-b36daea3035c","ICLR 2026杰出论文警示：LLM在多轮对话中平均性能暴跌39%","iclr-2026-multiturn-conversation-39pct-drop","2026-05-04T10:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"8c579d8d-4114-4d7d-92c1-22f5fab0537b","连续流语言模型跑到 8B：Lafa 单模型自验证解码提速 2.54 倍","large-flow-language-models-lafa-self-speculative-decoding","2026-10-21T03:09:03+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"da6cbb98-3b89-4631-8dbd-59411136fb30","哈佛用 700 家公司拆解 AI 编程智能体:30% 代码增量,49% 合并时延","harvard-coding-agent-review-bottleneck","2026-10-14T11:30:00+00:00"]