[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tb-mathmodel-agentic-verifiable-math-modeling-bench":3,"topics-all":38,"news-related-8857d393-a5f7-4b2a-8439-bf87ac324d4f":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"8857d393-a5f7-4b2a-8439-bf87ac324d4f","数学建模基准自曝:12个模型算对却只得49.82%","TB-MathModel 把 CUMCM\u002FMCM 真实竞赛建模题装进容器,用可执行评分器判分,再叠一层 624 条 Lean 4 义务验可行性。GPT-6 Astra 拿下 64.58%,2.54 美元的 DeepSeek 咬住 59.81%。审计还揪出参考答案本身物理不可行——是基准错了,不是模型。","数学建模是竞赛题，也是真实工作的接口层。\n\n一道刻意欠定的现实问题丢过来，你得自己定假设、选模型、写代码、跑求解器，最后交出一份站得住的结论。这条链上任何一环断了，答案就是废的。过去两年评测 Agent，测的几乎全是链条末端——代码能不能跑通、答案算得对不对。\n\n新发布的 TB-MathModel 把整条链塞进了评测里。任务本身并不刁钻，2024 年 CUMCM A 题的舞龙任务（dragon-dance），Agent 读题、看图、写代码，最后交出一份 JSON 算末端速度峰值。真正尴尬的是，这套评测在跑的过程中自己先摔了一跤。\n\n论文记录，12 个被测模型给出的峰值 1.246266 m\u002Fs 是对的，却因为参考答案本身算错而被压在 49.82%；另有 2 个模型跟着那份错答案走，反而拿了满分。错在哪：人类 O 奖论文对速度做了整数秒取整，跨过了 t=14.48 s 处的连续时间真实峰值，算出 1.406276 m\u002Fs，再乘上 1.604793 的放大系数变成 2.2568 m\u002Fs，越过了题面规定的 2 m\u002Fs 上限——这份\"标准答案\"物理上根本不可行。\n\n论文团队自己把这个问题挖出来并改了锚点值。论文写道：模型是对的，我们的锚点是错的。\n\n## 把\"建模\"变成机器能判的分\n\nTB-MathModel 收了 18 道真实竞赛建模题：CUMCM（中文）和 MCM（英文）2023—2025 每年 A\u002FB\u002FC 三题，各 9 道。题目原文、官方附件和从 PDF 里裁出来的图一起塞进 Docker 容器，Agent 要自己读题、开表格、看图、写代码、跑求解器，最后交一份 JSON。容器限制 2 核、8 GB 内存、无 GPU、4 小时超时、断网。\n\n关键在判分方式。论文把三类已有做法摆在一起对比：Mamo 有 1209 道题但只验 solver 片段；ModelingBench 68 道、MM-Bench 111 道都靠 LLM judge 给报告打分，论文对这类做法的评价是\"奖励流畅的散文，却不确认报出来的数字是否正确甚至是否站得住\"。TB-MathModel 只判最后一问的数字核心，用可执行 verifier 打分。\n\n还有两个设计比分数更值得看。\n\n一是**方向感知的单向打分**。每个指标显式标注 higher_is_better、lower_is_better、target_value 还是 exact_value，达到或超过锚点永远不会低于 1.0。锚定式打分的通病是：抄答案的拿高分，比答案更好的反而拿低分。单向打分把这条堵死了。\n\n二是**用 99 份人类 O 奖（Outstanding-award）论文做评审团**，而不是认准一份\"标准答案\"。99 份里有 94 份给出了被评的量，每题 1 到 18 份、中位数 4 份，一人一票。数学建模本就没有唯一解，把某一支队伍的选择当成 ground truth 本身就是错的。\n\n## 可行性交给 Lean 4 审卷\n\n更狠的一层在可行性校验。\n\n外层是程序化闸门，覆盖 18\u002F18 任务，把硬错误归进六类可判定的不变量形式：取值域与范围、整数与二值、序关系、精确恒等、有限域、划分求和。比如定日镜场那题要求额定功率不低于 60 MW、镜面安装高度在 2—6 米之间。\n\n里层是 Lean 4：把同样的不变量重新证一遍，18 道题一共生成 624 条内核检查的 obligation，不用 Mathlib，数值全部缩放成整数因此没有浮点容差；信任审计（#print axioms）只出现 propext 和 Quot.sound，没有 sorryAx，也没有 native_decide。\n\n这不是用 Lean 4 去解题——解题是模型的事。它是用来审 benchmark 自己的：闸门判的每一个决定，都要有一条机器可查的定理托底。\n\n闸门一共硬性作废 7 个格子，集中在定日镜场那题（4 个），另有烟幕、Juneau、奥运各 1 个。\n\n## 贵的不一定赢，加钱反而更差\n\n七个多模态组合、每题三个种子跑下来，O-Eval 是主指标，Multi-BO 是相对 DeepSeek-Flash 基线的增益指数（1.000 代表与基线持平，1.132 代表追平人类评审团剩余差距的 13.2%）：\n\n- GPT-6 Astra + codex：64.58%（±2.93），Multi-BO 1.132，246.41 美元\n- Fable 5.1 + claude-code：63.11%（±1.33），Multi-BO 1.081，272.80 美元\n- DeepSeek-Flash + opencode：59.81%（±1.25），Multi-BO 1.000，2.54 美元\n- GPT-5.5 xhigh + codex：59.75%（±1.55），Multi-BO 0.998，99.37 美元\n- Kimi K3 + opencode：58.23%（±1.12），Multi-BO 0.953，23.26 美元\n- GLM-5.3 + opencode：57.78%（±1.30），Multi-BO 0.941，28.15 美元\n- Qwen3.8 Flash + opencode：52.66%（±1.17），Multi-BO 0.803，8.30 美元\n\n两个结论。一是 2.54 美元的 DeepSeek V4.1 Flash 咬住了 59.81%，成本大约是头名的百分之一，落后约 4.8 分——这个性价比组合比单纯比谁分高更值得看。二是给 DeepSeek-Flash 调到最大推理档，多花的钱换来的是掉到 48.11%，Multi-BO 从 1.000 掉到 0.864。更长的思考预算在这些任务上不单调。\n\n还有一组数据专门回应\"分数到底是模型的还是壳子的\"：固定模型、只换四种 harness，O-Eval 全部加起来只动了 0.07 个百分点（59.81—59.88）。榜基本身测的是模型。\n\n不过作者也自己拆了台：种子间最大偏差到 2.93 个百分点，所以前两名的差距落在噪声里，不构成显著差异。而仓库里的实时榜单（https:\u002F\u002Fgithub.com\u002Fwxj630\u002Fterminal-bench-math-modeling）已经翻过一次——按单次跑分排，Fable 5.1 + claude-code 的 64.62% 反而压过 GPT-6 Astra + codex 的 61.47%。仓库自己写着：一两分的差距在种子噪声之内，没有模型越过 65%。\n\n## 所以呢\n\nTB-MathModel 真正的贡献不是又一张榜单，而是把\"可验证\"从赛题本身挪到了评测基础设施里：闸门查约束，Lean 4 查闸门，人类评审团查单一锚点的偏见。\n\n模型层越来越强，但只要评测还是 LLM judge 读报告、还是拿一份\"标准答案\"对齐，测出来的就只是流畅度和抄写能力。真正难的那层——把一团乱的现实压成一个能算、能验、能辩护的模型——目前测得最差。\n\n论文页上的话说得很直接（https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2611.08237）：离能处理真实世界问题的 Agent 还很远，而那条路需要的正是统一的环境、可验证的奖励，以及一份能复利增长的 agent 轨迹语料。据 AI Weekly 的报道（https:\u002F\u002Faiweekly.co\u002Falerts\u002Ftb-mathmodel-gpt-6-astra-tops-contest-bench-at-6458-o-eval），作者团队来自香港科技大学（广州）、IDEA Research 与 DataArcTech。\n\n这份基准自己也交了一次学费：错的是它。能让基准自己承认自己错的评测，才配拿来衡量模型。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08237","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"19bc7a09-aad7-4445-ba32-e6f47bad5812","en","A Math Modeling Benchmark That Blamed Itself","TB-MathModel scores agents on 18 modeling problems in containers, with an executable verifier and 624 Lean 4 obligations. GPT-6 Astra hit 64.58%.","Mathematical modeling is a competition problem, and also the interface layer of real work.\n\nA deliberately under-specified real-world problem shows up, and you have to set the assumptions yourself, pick a model, write the code, run the solver, and hand back a conclusion that holds up. If any link in that chain breaks, the answer is worthless. For the past two years, agent evaluation has mostly measured the tail end of that chain — whether the code runs, whether the number is right.\n\nThe new TB-MathModel puts the whole chain inside the evaluation. The tasks themselves are not exotic: the 2024 CUMCM A dragon-dance problem, where the agent reads the statement, looks at the figures, writes code, and returns one JSON artifact carrying the final speed peak. What is awkward is that the evaluation itself stumbled before the models did.\n\nThe paper records that 12 evaluated models reported the correct peak of 1.246266 m\u002Fs, yet were held to 49.82% because the reference answer itself was wrong. Two other models matched that erroneous reference value and scored 100%. Here is the error: the human Outstanding-award paper maximized over integer seconds and stepped over the true continuous-time peak at t=14.48 s, producing 1.406276 m\u002Fs; multiplied by the 1.604793 amplification ratio, that becomes 2.2568 m\u002Fs, above the 2 m\u002Fs cap stated in the problem. This \"correct answer\" was physically inadmissible.\n\nThe team dug the problem out themselves and corrected the anchor. The paper writes it plainly: the models were right and our anchor was wrong.\n\n## Turning modeling into something a machine can score\n\nTB-MathModel carries 18 real contest modeling problems: CUMCM (Chinese) and MCM (English), problems A, B and C for each of 2023 through 2025, nine each. The verbatim statements, official attachments, and figures cropped from the PDFs all ship into a Docker container. The agent has to read the statement, open the spreadsheets, look at the figures, write code, run the solver, and return one JSON artifact. The container is capped at 2 CPUs, 8 GB of memory, no GPU, a 4-hour timeout, with the network disabled.\n\nThe scoring method is the point. The paper lays three existing approaches side by side: Mamo has 1,209 tasks but only checks solver fragments; ModelingBench with 68 tasks and MM-Bench with 111 tasks both grade reports with an LLM judge, which the paper charges with \"rewarding fluent prose without establishing that the reported numbers are correct or even admissible\". TB-MathModel scores only the numeric core of the final question, using an executable verifier.\n\nTwo design choices matter more than the leaderboard does.\n\nThe first is **direction-aware, one-sided scoring**. Every metric explicitly declares whether it is higher_is_better, lower_is_better, target_value, or exact_value, and meeting or beating the anchor never scores below 1.0. Anchor-based scoring has a built-in pathology: copying the paper scores well, while beating the paper scores badly. One-sided scoring closes that hole.\n\nThe second is **a panel of 99 human Outstanding-award papers** rather than one designated \"correct answer\". Of the 99, 94 supply the quantity being scored; each task gets between 1 and 18 papers with a median of 4, and one paper equals one vote. Mathematical modeling admits no single right answer, so promoting one team's choices to ground truth is itself the error.\n\n## Feasibility handed to Lean 4 to grade\n\nThe sharper layer is feasibility verification.\n\nThe outer layer is a programmatic gate covering 18\u002F18 tasks, sorting hard errors into six decidable invariant forms: domain and range, integer and binary, ordering, exact identity, finite domain, and partition or sum. The heliostat field task, for instance, requires a rated power of at least 60 MW and mirror mounting heights between 2 and 6 meters.\n\nThe inner layer is Lean 4: it re-proves the same invariants as theorems, generating 624 kernel-checked obligations across the 18 tasks, with no Mathlib, and with all values scaled to integers so there is no floating-point tolerance. The trust audit (#print axioms) reports only propext and Quot.sound, with no sorryAx and no native_decide.\n\nThis is not using Lean 4 to solve anything — solving is the model's job. It is using Lean 4 to audit the benchmark itself: every decision the gate makes needs a machine-checkable theorem behind it.\n\nThe gate hard-invalidated 7 cells in total, concentrated on the heliostat task with 4, plus one each on smoke-screen, Juneau, and Olympic.\n\n## Expensive does not always win, and paying more can hurt\n\nAcross seven multimodal entries, three seeds per task. O-Eval is the headline metric; Multi-BO is the gain index against the DeepSeek-Flash baseline, where 1.000 means level with the baseline and 1.132 means closing 13.2% of the remaining gap to the human panel:\n\n- GPT-6 Astra + codex: 64.58% (±2.93), Multi-BO 1.132, $246.41\n- Fable 5.1 + claude-code: 63.11% (±1.33), Multi-BO 1.081, $272.80\n- DeepSeek-Flash + opencode: 59.81% (±1.25), Multi-BO 1.000, $2.54\n- GPT-5.5 xhigh + codex: 59.75% (±1.55), Multi-BO 0.998, $99.37\n- Kimi K3 + opencode: 58.23% (±1.12), Multi-BO 0.953, $23.26\n- GLM-5.3 + opencode: 57.78% (±1.30), Multi-BO 0.941, $28.15\n- Qwen3.8 Flash + opencode: 52.66% (±1.17), Multi-BO 0.803, $8.30\n\nTwo conclusions follow. First, the $2.54 DeepSeek V4.1 Flash run holds 59.81% at roughly one hundredth the leader's cost, trailing by about 4.8 points — that cost-performance ratio is more worth watching than any score comparison. Second, pushing DeepSeek-Flash to its maximum reasoning tier made things worse: the multi-reference score fell to 48.11% and Multi-BO dropped from 1.000 to 0.864. A longer deliberation budget is not monotone on these tasks.\n\nThere is also a set of numbers aimed squarely at the question of whether the board measures the model or the harness: with the model fixed and only the harness varied across four options, O-Eval moved by at most 0.07 percentage points in total (59.81 to 59.88). The board is measuring the model.\n\nThe authors also undercut their own headline, which is to their credit. Per-seed deviations reach 2.93 percentage points, so the gap between the top two sits inside noise and does not constitute a significant difference. And the live leaderboard in the repository (https:\u002F\u002Fgithub.com\u002Fwxj630\u002Fterminal-bench-math-modeling) has already flipped once — on single runs, Fable 5.1 + claude-code at 64.62% actually edges out GPT-6 Astra + codex at 61.47%. The repository itself notes that a one-to-two point gap is inside seed noise, and that no model clears 65%.\n\n## So what\n\nThe real contribution of TB-MathModel is not another leaderboard. It is moving \"verifiable\" out of the contest problems and into the evaluation infrastructure: the gate checks constraints, Lean 4 checks the gate, and the human panel checks the bias of any single anchor.\n\nThe model layer keeps getting stronger. But as long as evaluation is an LLM judge reading a report, or aligning against one designated \"correct answer\", what you are actually measuring is fluency and transcription. The genuinely hard layer — compressing a messy real-world situation into a model that is computable, verifiable, and defensible — remains the least adequately measured one.\n\nIts words on the Hugging Face paper page (https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2611.08237) are blunt: agents that can handle real-world problems are still a long way off, and the road there needs exactly a uniform environment, a verifiable reward, and an agent-trajectory corpus that compounds. According to AI Weekly (https:\u002F\u002Faiweekly.co\u002Falerts\u002Ftb-mathmodel-gpt-6-astra-tops-contest-bench-at-6458-o-eval), the authors come from The Hong Kong University of Science and Technology (Guangzhou), IDEA Research and DataArcTech.\n\nThis benchmark paid its own tuition first: it was the one that was wrong. An evaluation willing to say that out loud is the only kind worth measuring models with.","tb-mathmodel-agentic-verifiable-math-modeling-bench","2026-11-08T03:15:00Z","2026-11-08T03:14:31.128083Z","2026-11-08T03:14:31.128098Z",true,"agent",45,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"37e0486a-225d-4b44-b189-206bd756210c","多轮 Agent 强化学习提升了什么:46.8% 的真相","multi-turn-agentic-rl-budget-dependence","2026-11-06T01:18:33+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"99c1eb7b-9226-4d61-afdb-5d54f920a85c","研究品味实测:AI 智能体的想法匹配只有人类的 18%","ideataste-llm-research-taste-benchmark","2026-11-04T11:13:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"a50b0336-dabe-405e-b8c4-d27156ab6d24","60% 权重转私有测试集:开放权重落后榜首 12 分","artificial-analysis-intelligence-index-v5","2026-10-28T07:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"ba7f04d1-2fe5-4e1d-90a6-94ec3c4b70df","阿里 UserArena:64,729 条购物轨迹当考场","userarena-shopping-agent-user-simulation-benchmark","2026-10-26T09:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"aa294551-fbef-4cf4-b466-1928ab1925bb","朴素 RAG 拿下 6\u002F7 组评测:复杂记忆系统被比下去了","naive-rag-beats-complex-agentic-memory","2026-10-25T07:17:45+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"da6cbb98-3b89-4631-8dbd-59411136fb30","哈佛用 700 家公司拆解 AI 编程智能体:30% 代码增量,49% 合并时延","harvard-coding-agent-review-bottleneck","2026-10-14T11:30:00+00:00"]