[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-mmdm-decision-model-benchmark":3,"topics-all":38,"news-related-f2723003-43ac-47fe-bb59-49030446919a":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"f2723003-43ac-47fe-bb59-49030446919a","28个决策模型大考:多选最高才36.81%","港中大与港科大的 MMDM 基准用 5,448 道新题、16 类任务考了 28 个决策模型:单选最高 76%,多选最高只有 36.81%,微软 Decision-1 约束规划仅 13.5。知识题普遍会做,选项联动、约束感知和完整多选恢复是全行业短板,选对答案也不代表概率可用,上线前值得先对准自身负载。","9 月中旬,Jev 把「决策模型」做成一个新品类:不写文章,一次前向直接吐出每个选项的概率。两个月里这个赛道挤进了 28 个能拉出来同台对比的型号——闭源的 Jev 1.14.0、微软 Decision-1、GPT-6 Luna,开源的 WeMM-JEV、KnowLine、Perplexity Decider、Clef 全部到场。11 月 10 日,港中大与港科大团队在 arXiv 放出新考卷 MMDM([arXiv:2611.14309](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14309)),5,448 道题把 28 个模型排进同一张表——结果和各家发布时的口径,对不上了。\n\n## 一张防背题的考卷\n\nMMDM 的 5,448 道题覆盖 16 类任务:世界知识、阅读理解、演绎与量化推理、agent 状态跟踪、工具与技能选择、偏好判断、rubric 打分。题型分三种:4,112 道硬单选(唯一正确项),633 道硬多选(要求给出完整有效集合),703 道软标签(要求匹配整条概率分布)。选项数从 2 个到 1,024 个,其中 94 道题超过 255 个选项;完整输入用 Qwen3.5-4B 分词器度量,中位数 827 token,最长 64,817。\n\n题目全部重新构造:改写既有公开数据集但保留原答案、按源数据重组新题、全新出题,再过五模型多数筛查加独立人工复审,外加一道只给选项不给题面的探针,专防模型走捷径。干扰项也不是随手编的,全部围绕可识别的失败模式设计:套用已过期的规则、混淆相似实体、忽视动作的前置条件。动机写在第一节:既有决策评测大量复用公开基准,测试集污染是首要担忧;论文对比的几个既有决策基准里,多选题一栏全部空缺。\n\n## 分数单:知识都会,联动全崩\n\n单选总分最高的是开源的 WeMM-JEV 27B,76.00 分;闭源最高 Liquid-d1-Ultra 73.32;微软 Decision-1 单选 61.36,闭源组排第四。它的发布口径是另一回事——据 The Decoder 报道,微软自报在自家 36 项基准、约 15 万道题的对比里平均准确率 83.5% 排第一([来源](https:\u002F\u002Fthe-decoder.com\u002Fmicrosofts-decision-1-model-enters-the-fast-growing-ai-decision-model-race\u002F))。换一张没背过的卷子,分数就是分数。\n\n类别拆开更扎眼:Perplexity Decider-V1.2 世界知识 78.75,选项逻辑 53.95,约束规划 47.95——读得懂世界,算不清选项之间的依赖。微软 Decision-1 的约束规划只有 13.5,除五个 4B 及更小的模型外全场垫底。多选则是集体灾难:全场最高的多选准确率 36.81,仍出自 WeMM-JEV;衡量「把有效集合完整捞出并合理分配概率」的 SB-F 指标,28 个模型没有一个达到 0.20。\n\n## 选对答案,不等于概率可用\n\n决策模型真正的卖点是被当作概率源使用,论文专门拆了这一层。WeMM-JEV 和 KnowLine 31B 单选只差 0.93 个百分点(76.00 对 75.07),KnowLine 的 Soft Brier 0.1233 却约为 WeMM-JEV 0.2327 的一半,Score nMAE 0.0624 不到对方 0.2203 的三分之一。校准同样分层:五个代表模型里微软 Decision-1 的 ECE 最大,中高置信段整体偏乐观;OpenJev 最高置信桶平均置信 0.97,实际命中 0.87。选型只看「选对率」,下游拿概率设阈值的管线就是在踩雷。\n\n## 挑模型看类别,别只看总分\n\n类别冠军和总分冠军常常不是同一个:证据校验最高分属于 Quyet-1.0-Large 的 89.1,情绪识别最高是 Mapika Decider 4B 的 64.0。路由能力也不通用:GPT-6 Luna Decisions 意图路由 65.3 压过 Jev 的 63.4 和微软的 62.6,工作流决策却只有 16.8,远低于两者的 52.2 和 43.5。总分接近的模型长处可能完全错开——不先看清自己的负载落在哪一列,选型就近乎抽奖。\n\n## 所以呢\n\n决策模型这一波要的不是「更聪明」,而是「快、便宜、输出直接接管线」。但 MMDM 的数据说明,概率质量还没跟上铺货速度:多选恢复、约束感知、选项联动,全行业一起没做对。数据集已开在 Hugging Face,代码与完整榜单挂在项目页。上线之前,先把自己的工作负载对进 16 列里挑模型,比追总分榜实在得多。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14309","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"8c98b79b-184e-488d-a5e0-4e4ace4c683e","en","28 Decision Models Tested: Best Multi-Select Is Just 36.81%","MMDM runs 28 decision models through 5,448 new questions: top single-select 76%, multi-select 36.81%, Decision-1 13.5% on constraint planning.","In mid-September, Jev turned the \"decision model\" into a product category: no prose, just one forward pass that outputs a probability for every option. Two months later the field holds 28 models worth comparing side by side — closed-source Jev 1.14.0, Microsoft Decision-1 and GPT-6 Luna, alongside open-source WeMM-JEV, KnowLine, Perplexity Decider and Clef. On November 10, a team from CUHK and HKUST posted a new exam to arXiv, MMDM ([arXiv:2611.14309](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14309)), which puts all 28 models on one scoreboard — and the results no longer match the launch-day claims.\n\n## An exam built against memorization\n\nMMDM's 5,448 questions span 16 task categories: world knowledge, reading comprehension, deductive and quantitative reasoning, agent state tracking, tool and skill selection, preference judgment, and rubric scoring. There are three answer types: 4,112 hard single-select questions (one correct option), 633 hard multi-select questions (the complete valid set is required), and 703 soft-label questions (the full probability distribution must be matched). Option sets run from 2 to 1,024 options, with 94 questions exceeding 255; measured with the Qwen3.5-4B tokenizer, complete inputs have a median of 827 tokens and a maximum of 64,817.\n\nEvery question is freshly constructed: existing public datasets are rewritten while keeping their original answers, new questions are recomposed from source data, and entirely new tasks are written — then screened by a five-model majority vote plus independent human expert review, with an option-only probe (options shown, state and query hidden) to catch shortcut answers. Distractors are not thrown together either: they are designed around identifiable failure modes such as applying an expired rule, confusing similar entities, or overlooking an action's prerequisite. The motivation is stated up front: existing decision-model evaluations reuse public benchmarks heavily, making test-set contamination a first-order concern; and among the prior decision benchmarks compared in the paper, the multi-select column is empty across the board.\n\n## The scoreboard: knowledge easy, joint reasoning not\n\nThe best single-select score belongs to open-source WeMM-JEV 27B at 76.00; the best closed-source model, Liquid-d1-Ultra, reaches 73.32; Microsoft Decision-1 sits at 61.36, fourth among closed-source entries. Its launch framing told a different story — according to The Decoder, Microsoft reported 83.5% average accuracy and first place in its own 36-benchmark comparison of roughly 150,000 questions ([source](https:\u002F\u002Fthe-decoder.com\u002Fmicrosofts-decision-1-model-enters-the-fast-growing-ai-decision-model-race\u002F)). On an exam the models have not seen, the score is the score.\n\nBroken out by category it gets worse. Perplexity Decider-V1.2 scores 78.75 on world knowledge but 53.95 on option logic and 47.95 on constraint planning — it reads the world fine but cannot work out dependencies between options. Microsoft Decision-1 manages just 13.5 on constraint planning, ahead of only five models at 4B or smaller. Multi-select is a collective failure: the best accuracy across all 28 models is 36.81, again WeMM-JEV; on SB-F, which measures whether a model recovers the complete valid set and distributes probability over it sensibly, not a single model reaches 0.20.\n\n## Picking the right answer is not the same as usable probabilities\n\nThe real selling point of decision models is being used as a probability source, and the paper isolates that layer. WeMM-JEV and KnowLine 31B differ by 0.93 points in single-select accuracy (76.00 vs 75.07), yet KnowLine's Soft Brier of 0.1233 is about half of WeMM-JEV's 0.2327, and its Score nMAE of 0.0624 is less than a third of 0.2203. Calibration also stratifies: among five representative models, Microsoft Decision-1 has the largest ECE and stays overconfident through the mid-to-high range, while OpenJev's top confidence bin averages 0.97 confidence against 0.87 accuracy. Select on \"how often it picks correctly\" alone, and any downstream pipeline that thresholds on the probabilities is walking into a trap.\n\n## Pick by category, not by total score\n\nCategory champions are often not the overall champions: the best evidence-validation score belongs to Quyet-1.0-Large at 89.1, and the best emotion recognition to Mapika Decider 4B at 64.0. Routing skill does not transfer either: GPT-6 Luna Decisions beats Jev (65.3 vs 63.4) and Microsoft (62.6) on intent routing, yet scores only 16.8 on workflow decisions, far below their 52.2 and 43.5. Models with near-identical totals can have completely disjoint strengths — without checking which column your workload lives in, model selection is close to a lottery.\n\n## So what\n\nThis wave of decision models is not about being smarter; it is about being fast, cheap, and pluggable into pipelines. But MMDM's numbers say probability quality has not kept pace with the shipping speed: multi-select recovery, constraint awareness, and option interdependency are unsolved across the entire field. The dataset is open on Hugging Face, with code and the full leaderboard on the project page. Before deploying, map your own workload onto those 16 columns and choose accordingly — that beats chasing the total-score leaderboard by a wide margin.","mmdm-decision-model-benchmark","2026-11-12T13:30:00Z","2026-11-12T13:12:16.391830Z","2026-11-12T13:12:16.391841Z",true,"agent",132,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"6c8ae96a-53e0-4ef7-8609-40b345cdee7c","决策模型实测:自己演砸42.9%,教学生82.9%","decision-model-actor-judge-teacher","2026-11-15T21:45:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"bd350599-7253-4f32-a874-9224bab89a49","多智能体搜索实测：32组对照里委派多数时候反而更差","multi-agent-search-delegation-accuracy","2026-11-15T09:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"a2038f05-8ad2-4b57-bb29-47ff522e09a0","失败轨迹偷走权重:ALFWorld成功率涨7到9.4点","outcome-mass-alignment-multi-turn-agent-rl","2026-11-14T11:20:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"9531b4f7-3f4d-474c-b233-c715925a3e76","8.4B轨迹压过25.3B技能语料:形式化证明中训配方","davinci-prover-agent-native-mid-training","2026-11-14T07:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"343f6cae-ae14-4617-a317-7316748e1d6e","删掉推理链再转发:同档32B模型差51分","deletion-sensitivity-reasoning-trace-routing","2026-11-13T07:05:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"365a0af2-d564-4eff-bf03-a138cdea0c5f","研究智能体大考:只查交付过69%,查全要求剩28%","are-bench-research-agent-context-requirements","2026-11-13T01:00:00+00:00"]