[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sp3o-sparse-critic-supervision":3,"topics-all":38,"news-related-176b4807-da61-479f-a514-9381cd13319e":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"176b4807-da61-479f-a514-9381cd13319e","SP3O:3 个锚点修复 PPO critic 的平坦化","上海交大与上海AI Lab 等机构发现 PPO critic 的系统性失效 Value Flattening:MC 价值在响应内剧变,预测却近乎平线。SP3O 每条响应只监督 3 个状态,Qwen3-4B\u002F8B 数学与泛化平均分超 PPO 和 GRPO。","PPO 在大模型强化学习里常用一个 critic 估计状态价值、给策略更新降方差。上海交通大学、上海 AI Laboratory 等机构的联合团队最近发现,critic 存在一个此前未被命名的系统性失效模式,并给出一个改动量极小的修复。论文《Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening》发布于 arXiv(2609.18708),登上 Hugging Face 日榜第三。\n\n## Value Flattening:critic 在「装平」\n\n团队用多条蒙特卡洛(MC)续算估计一条推理响应中每个中间状态的真实价值,再对比 critic 预测:MC 价值剧烈起伏,critic 预测却近乎平线。团队将其命名为 Value Flattening。现象在受控的 FrozenLake 环境同样复现,且状态空间越大越明显——token 动辄上千的 LLM 推理因此格外严重。\n\n## 病根有两层\n\n- **隐式方差惩罚**:只有终局奖励、γ=λ=1 时,响应内所有状态共享同一个采样终局回报。对稠密 MSE 损失做代数分解,会分离出一项直接惩罚「响应内预测方差」——对 critic,把预测压平就是最小化损失的最省事解。\n- **冗余的邻近更新**:LLM 相邻两个状态只差一个 token,共享几乎全部历史,表征与梯度高度相似;稠密 token 级监督等于把几乎同一个更新重复几百遍,信息没增加,方差惩罚的副作用却在累积。\n\n## SP³O:每条响应只监督 3 个状态\n\n修复方案 SParse Proximal Policy Optimization(SP³O):actor 目标、rollout 流程、return 目标全部不变,只把 critic 损失从逐 token 稠密计算,改为每条响应只施加在少数几个分隔良好的状态上,3 个即可。消融显示:锚点固定在响应进度 30%\u002F60%\u002F90% 处平均 45.57%,优于随机放置的 36.59%(稠密 PPO 基线 37.60%);3-8 个监督点的稀疏配置优于更密变体,监督越密,结果越向稠密基线回落。\n\n## 跑分(团队自报)\n\n在 Qwen3-4B-Base 与 Qwen3-8B-Base 上(域内用 DAPO-Math-17k 训练,数学基准 32 次生成取平均):4B 七项数学基准平均 45.57%,高于 GRPO 的 39.26% 与 PPO 的 37.60%,AIME26 从 PPO 的 14.69% 提到 22.08%;8B 平均 50.51%,同样居首。泛化评测(ARC-C、MMLU-Pro、GPQA 等,4 次生成取平均)4B\u002F8B 平均 59.28% 与 66.37%,均为三者最高。需要说明并非逐项全胜:8B 的 Minerva 一项 GRPO 为 52.06%、高于 SP³O 的 47.40%;8B 泛化侧 ARC-C 与 MMLU-Pro 也分别由 PPO(93.84%)与 GRPO(66.27%)领先。SP³O 优势在平均分。\n\n## 所以呢\n\n代码基于 THUDM\u002Fslime v0.2.4 构建,Apache-2.0 开源,README 的 4B 复现脚本是 8 卡配置——对正在跑 PPO 类训练的团队,这是近乎零迁移成本的 critic 侧改动,值得直接 A\u002FB。对比基线里 GRPO 已是默认参照,而本文走的是另一条路——不绕开 critic,而是承认稠密监督对它不友好,把监督本身当稀缺资源来设计。RLHF 的下一次涨分,未必来自更大的奖励模型,也可能来自对既有组件更细的拆解。\n\n参考:arXiv 2609.18708;github.com\u002FDodojordi\u002FSP3O","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.18708","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"f86c65fb-3186-46fb-94cc-d588294bea71","en","SP3O fixes PPO critic flattening with 3 supervised states","Team finds PPO critics go flat while MC values swing; SP3O supervises just 3 states, topping PPO and GRPO averages on Qwen3 4B\u002F8B.","PPO in large-model reinforcement learning commonly uses a critic to estimate state values and reduce the variance of policy updates. A joint team from Shanghai Jiao Tong University, Shanghai AI Laboratory and five other institutions recently gave this component a systematic checkup: they identify a previously unnamed systematic failure mode in the critic, and propose a fix whose change is surprisingly small. The paper, \"Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening\", is on arXiv (2609.18708) and reached #3 on Hugging Face Daily Papers.\n\n## Value Flattening: critics play flat\n\nThe team estimates the true value of every intermediate state in a reasoning response using multiple Monte Carlo (MC) continuations, then compares those with PPO critic predictions. MC values swing sharply within a response, while critic predictions stay close to a flat line. The team names this phenomenon Value Flattening. It reproduces in a controlled FrozenLake environment and grows more pronounced as the state space expands — which explains why it bites especially hard on LLM reasoning, where responses span thousands of tokens.\n\n## Two root causes\n\n- **Implicit variance penalty.** Under the common setting of terminal-only rewards with γ = λ = 1, every state in a response shares the same sampled terminal return. Decomposing the dense MSE loss yields a term that directly penalizes within-response prediction variance — for the critic, flattening predictions is the cheapest way to minimize the loss.\n- **Redundant neighboring updates.** Two adjacent LLM states differ by a single token and share almost their entire history, so their representations and gradients are highly similar. Dense token-level supervision repeats nearly the same update hundreds of times: no new information gets in, while the variance-penalty side effect accumulates.\n\n## SP³O: supervise three states per response\n\nThe fix, SParse Proximal Policy Optimization (SP³O), keeps the actor objective, rollout procedure and return targets untouched, and applies the critic loss only at a few well-separated states per response. The paper reports that supervising three states per response suffices. The ablation shows that anchors fixed at 30%\u002F60%\u002F90% response progress average 45.57%, clearly above random placement at 36.59% (dense PPO baseline: 37.60%); sparse configurations with 3-8 supervised states beat denser variants, and the denser the supervision, the closer results fall back toward the dense PPO baseline.\n\n## Benchmarks (self-reported)\n\nOn Qwen3-4B-Base and Qwen3-8B-Base (in-domain trained on DAPO-Math-17k; math benchmarks averaged over 32 generations): the 4B model averages 45.57% across seven math benchmarks, above GRPO's 39.26% and PPO's 37.60%, with AIME26 rising from PPO's 14.69% to 22.08%; the 8B model averages 50.51%, also first. On out-of-distribution suites (ARC-C, MMLU-Pro, GPQA and others, averaged over four generations), the averages are 59.28% and 66.37%, both the highest of the three methods. To be fair, it is not a clean sweep: on 8B Minerva, GRPO scores 52.06% versus SP³O's 47.40%; on the 8B out-of-distribution suite, ARC-C and MMLU-Pro are led by PPO (93.84%) and GRPO (66.27%) respectively. SP³O's edge is on the average.\n\n## So what\n\nThe code is built on THUDM\u002Fslime v0.2.4, open-sourced under Apache-2.0, with an eight-GPU script for the 4B run — for teams already training with PPO, this is a near-zero-migration critic-side change worth A\u002FB testing directly. The more interesting part is the signal: GRPO already sits in the comparison table as the default reference, and this paper demonstrates the other path — not sidestepping the critic, but accepting that dense supervision is unfriendly to it, and treating supervision itself as a scarce resource to design around. The next RLHF gain may come not from a larger reward model, but from a finer dissection of the components we already have.\n\nReferences: arXiv 2609.18708; github.com\u002FDodojordi\u002FSP3O","sp3o-sparse-critic-supervision","2026-09-17T17:10:01Z","2026-09-17T17:10:09.415898Z","2026-09-17T17:10:09.415907Z",true,"agent",2,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"d41175a7-ad10-4e00-9017-a148fa0a77b3","BenchMIRT 把 LLM 基准拆到单题:Ai2 想让模型排名不再「一张考卷定生死」","ai2-benchmirt-llm-benchmark-audit","2026-09-10T11:05:05+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"5bfdf32b-44eb-4eb5-a98b-39e921168182","九天内连发五款前沿模型:7 月的大模型军备赛,真正决胜负的不再是 benchmark","july-2026-five-frontier-models","2026-07-23T12:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"8173a86b-4e5e-429a-8ddf-f98af527b4b5","LLM-as-a-Verifier：验证成 LLM 第四 scaling 维度","llm-as-a-verifier-fourth-scaling","2026-07-07T12:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"ec2c558c-502d-43a5-9494-c766dfd515e9","EurekAgent：把科学发现的瓶颈从「工作流」拽到「环境」，11 美元跑出 26 圆 packing 新 SOTA","eurekagent-environment-engineering-11-usd","2026-06-11T17:56:35+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"af09e362-6537-4b62-bf46-8c8c4ce00982","2026 AI Index报告：开源与闭源LLM差距为何重新拉大？","stanford-ai-index-2026-open-vs-closed-3pct","2026-05-30T04:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"6f1f105b-8e80-4b2c-b88c-b392556952aa","2026年本地LLM深度评测：开源模型性能全解析","local-llm-2026-deep-eval-swe-bench-aime","2026-04-25T11:15:00+00:00"]