[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-grpo-dense-rewards-matched-compute":3,"topics-all":38,"news-related-a6d8e95f-41ba-4a97-815c-94eff9401706":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"a6d8e95f-41ba-4a97-815c-94eff9401706","按步数赢按算力输:GRPO密集奖励的评测账本","论文用 1,668 次训练、六个模型做预注册对照，发现密集过程奖励在 GRPO 上按训练步数比领先 1.5 分，换成同等 token 预算却落后 14.3 分——验证用的续写烧掉了 91% 的生成 token。真正稳的做法是挑题而不是改奖励，基线学习率翻倍就能抹平大部分差距，这条纪律值得做 RL 训练的人记住。","RL 训练这两年流行一类「过程奖励」：不看最终答案对不对，给推理链每一步单独打分，理由是模型能少走弯路。评估的默认动作是跟只看最终答案的 GRPO 在**同样训练步数**下比谁准——只数步数，不数账单。\n\n一篇 11 月 10 日提交到 arXiv 的论文（[arXiv:2611.13560](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13560)，19 位作者，正文 10 页、全文 59 页，含 18 图 27 表）拆了这条默认路径。结论一句话：把算力算进去，排名直接翻转。\n\n## 密集奖励的账单：91% 的 token 花在废稿上\n\n蒙特卡洛过程奖励的做法是：在一条推理链的 5 个固定位置切一刀，切成 6 段，每刀从该前缀采样 4 条续写，看有多少条能续到正确答案，据此估计前缀的价值。问题是这些续写只用来打分，打完就丢，从不进入训练。\n\nk=4 时，续写吃掉 91% 的全部生成 token。同样一份预算，GRPO 能跑 584 次更新，密集奖励只能跑 50 次。\n\nGPU 时间更不成比例，因为续写由推理引擎大批量生成，单 token 比训练一步便宜。同样 token 预算下，H200 上密集奖励训 0.14 小时、GRPO 训 0.43 小时。论文的分解：GRPO 把 61% 时间花在更新上，密集奖励把 73% 花在生成续写上。\n\n## 排名是怎么翻的\n\n按训练步数比密集奖励是赢的：每步的梯度效率是纯 GRPO 的 1.56 倍 [1.28, 2.03]，同步数领先 1.5 分 [1.0, 2.1]。\n\n换成同等 token 预算，它落后 14.3 分 [12.7, 16.0]（n=5，Holm 校正 p=5×10⁻⁵），12 组全新种子上落后 14.9 分 [14.0, 15.8]。同等 GPU 时间落后 7.8 分 [6.8, 8.8]，12 组种子一致。完成长度放宽到 768 token，差距缩到 6.6 分但方向不变；在 Qwen3 的 0.6B、1.7B、4B 上分别落后 8.0、9.1、10.0 分，GSM8K 上落后 9.8 分。\n\n论文还做了更硬的一步：给密集奖励和基线同样长的训练时间（1.29 A100 GPU 小时，8 组种子，严格验证器）。四个变体没有一个显著超过基线——k=4 蒙特卡洛落后 3.3 分 [2.4, 4.3]，复用续写变体落后 5.2 分 [4.1, 6.3]，VinePPO 式变体只落后 1.1 分 [0.4, 1.7]，落在 ±2 分等效区间内（TOST p=.005）；k=1 落后 1.1 分 [−0.5, +2.8]，既不显著也未被证明等效。\n\n## 便宜的验证方法会骗人\n\n论文还解剖了另一个流传更广的替代验证法：梯度测试——在固定 batch 上看奖励的更新方向与「准确率梯度」的独立估计是否同向。蒙特卡洛奖励确实同向，平均余弦 0.84。但真训起来，低权重（α=0.25\u002F0.5\u002F1）加进去与纯 GRPO 在 ±3 分内等效，高权重（α=5）直接掉 12.4 分 [8.7, 16.1]。\n\n更微妙的是，训练崩掉时对齐度纹丝不动（0.969 到 1.000），同期熵从 0.21 涨到 0.58，相对初始策略的 KL 约为低权重跑的 30 倍——不是奖励指错方向，是被过度优化。\n\n所以梯度测试能排序，不能定量。每 token 效率测试复现了这个排序（k=1 是 0.45 倍、k=2 是 0.27 倍、k=4 是 0.15 倍纯 GRPO），但对五个被测奖励都高估 0.8 到 1.8 分——高估幅度和它想测出的收益一样大。\n\n## 真正有用的那一半：挑对题，省五分之一预算\n\n另一半反例是提示选择：不改奖励，只在生成前花一次前向 pass 挑题。每轮抽 24 个候选题，用策略隐状态上的 ridge critic 预测每题成功率，挑最接近 1\u002F2 的 6 个，再随机加 2 个。\n\n效果比密集奖励好：Qwen3-0.6B 上 +1.7 分 [1.2, 2.1]（54 组种子；24 组全新种子的预注册复现 +1.2 分，Holm p=.003），Qwen2.5-1.5B 上 +0.8 分 [0.4, 1.1]；换到评测集之外的 1,543 道 MATH 题仍成立（+0.8 和 +1.1），在两配置都没训过的 GSM8K 上 +1.1 和 +1.3 分（次要指标）。算上训练时间仍领先 1.0 分和 0.5 分。\n\n作者还加了随机对照：同样 24 道题里随机挑。随机选比过滤器低 1.7 分 [0.5, 2.9]（p=.008），说明收益来自「挑了哪道题」而非「更快刷过题库」。\n\n收益量级也说得很保守：峰值 2.6 分 [2.2, 3.0] 出现在预算的十六分之三处，到预算末尾只剩 0.8 分；训到 2 倍预算时优势缩到 0.5 分 [−0.5, +1.6]，落在 ±2 分等效区间（TOST p=.004）。换算下来，过滤器用 0.78 的 token 预算就追上了纯 GRPO 的终局精度。\n\n更扎人的是：把基线的常数学习率翻倍，GRPO 自己涨 1.9 分 [0.5, 3.3]（Holm p=.018），与过滤器的差只剩 0.03 分 [−1.3, +1.2]（等效检验 p=.002）。一个「新方法」的收益，大半是基线没调好。\n\n## 论文自己承认的边界\n\n这类论文最该看局限章节，作者列得坦白：\n\n- 六个模型里四个来自 Qwen 家族，Llama-3.2 只测了提示选择、只测了一个题池；论文明说「没有在 Qwen 家族之外展示过收益」。\n- 「候选题成功率低于 1\u002F2 就有效」这条规律是**事后**从五个设置里总结的，六个预注册测试对了五个；唯一预测会涨的 Llama-3.2-1B 没兑现（+0.4 分 [−0.1, +1.0]）。\n- 预注册计划文件是内部文件，不在公开注册表里；部分主要对比只有 3 到 5 组种子。\n- 宽松的验证器会自己漂移。默认验证器把「最后一个数字」也算对时，部分训练跑着跑着就不再把答案装进 `boxed` 里，一次研究 408 次评测里最多 49 次漂移；改用只认 `boxed` 的严格验证器重训后漂移消失（2,196 次评测 0 次）且不损精度。\n\n## 所以呢\n\n这篇论文真正该带走的不是「密集奖励没用」——作者自己也承认，在同等训练时间口径下 k=1 变体既没显著输、也没被证明等效，结论并不干脆。真正该带走的是**一条评测纪律**：看到「新奖励让模型涨了 X 分」，先问三个问题——按训练步数算的还是按 token 算的？算 token 时，验证用的续写进账单了吗？基线学习率调过没有？\n\n论文结论里的措辞很克制：宣称某个训练干预有效时，应该同时报出 token 预算、GPU 小时预算、验证器口径、在哪些模型上复现过，以及基线怎么调的。少报一项，结论可能差 14 分。\n\n论文声明会附带训练与评测设置、统计方法、预注册计划和逐次运行记录（[arXiv 全文 HTML](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2611.13560v1)、[alphaXiv 摘要页](https:\u002F\u002Fwww.alphaxiv.org\u002Fabs\u002F2611.13560)）。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13560","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"5c3d3b83-a8ff-404b-9716-fb4ed6b2f3a7","en","Dense Rewards Win Per Update, Lose Per Token: A GRPO Audit","Across 1,668 runs on six models, dense GRPO rewards win per update but trail 14.3 points at matched tokens: rollouts eat 91%. Prompt selection wins.","Reinforcement learning has spent the last two years on a family of \"dense rewards\": instead of checking only whether the final answer is right, you score each step of the reasoning chain separately, on the theory that a model which gets partial credit for partial progress will make fewer wrong turns. The default way to evaluate such a method is to train it against outcome-only GRPO for the same number of updates and see which one ends up more accurate. That comparison counts steps but never counts the bill.\n\nA paper submitted to arXiv on 10 November ([arXiv:2611.13560](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13560), 19 authors at Case Western Reserve University, 59 pages with 10 pages of main text, 18 figures and 27 tables) takes that default path apart. Its conclusion fits in one line: once compute is counted, the ranking reverses.\n\n## The bill for dense rewards: 91% of tokens go to discarded drafts\n\nThe Monte-Carlo process reward works like this: cut a completion at five fixed token positions, which splits it into six segments, and at each cut sample four continuations from that prefix; the fraction of continuations that reach the correct answer estimates how promising the prefix is. The problem is that these continuations are only scored and then thrown away — they never enter training.\n\nAt k=4, those continuations consume 91% of all generated tokens. On the same token budget, outcome-only GRPO gets to run 584 updates; the dense reward gets 50.\n\nGPU time is worse, because the ratio is not proportional. Continuations are produced by the inference engine in large batches, which is much cheaper per token than a training step. On the same token budget on an H200, the k=4 dense reward trains for 0.14 hours while outcome-only GRPO trains for 0.43. The paper's own breakdown: outcome-only GRPO spends 61% of its time on updates, while the Monte-Carlo reward spends 73% of its time generating continuations.\n\n## How the ranking flips\n\nPer update, the dense reward is winning. Its gradient efficiency is 1.56x [1.28, 2.03] that of outcome-only GRPO per update, and it leads by 1.5 points [1.0, 2.1] at an equal number of updates.\n\nSwitch to an equal token budget and it falls 14.3 points behind [12.7, 16.0] (n=5, Holm-corrected p=5x10^-5), and 14.9 points behind [14.0, 15.8] on twelve fresh seeds. At equal training time it trails by 7.8 points [6.8, 8.8], in all twelve seeds. Longer completions (768 tokens) shrink the gap to 6.6 points but do not flip its sign; the deficit recurs on Qwen3 0.6B, 1.7B and 4B at 8.0, 9.1 and 10.0 points, and on GSM8K at 9.8 points.\n\nThe paper then takes the harder step: give the dense rewards the same training time as the baseline (1.29 A100 GPU-hours, 8 seeds, strict verifier). None of the four variants significantly beats the baseline. The k=4 Monte-Carlo reward trails by 3.3 points [2.4, 4.3]; the variant that also trains on its continuations trails by 5.2 [4.1, 6.3]; the VinePPO-style variant trails by only 1.1 [0.4, 1.7], a deficit that sits inside the +\u002F-2 point equivalence margin (TOST p=.005); and the k=1 reward trails by 1.1 [-0.5, +2.8], neither significant nor shown to be equivalent.\n\n## Why the cheap validation method misleads\n\nThe paper also takes apart a more widely used substitute for a full comparison: the gradient test, which asks whether a reward's update points in the same direction as an independent estimate of the gradient of expected accuracy. The Monte-Carlo reward does align — mean cosine 0.84. But once you actually train with it, adding it at a low weight (alpha = 0.25, 0.5 or 1) is equivalent to outcome-only GRPO within +\u002F-3 points, and adding it at a high weight (alpha = 5) loses 12.4 points [8.7, 16.1].\n\nThe subtler part is that the alignment never decays while training collapses. It stays at 0.969 to 1.000, while entropy climbs from 0.21 to 0.58 and the KL from the initial policy reaches about 30 times that of the low-weight runs. The reward is not pointing the wrong way; it is being over-optimized.\n\nSo the gradient test can rank, but it cannot size. The per-token efficiency test reproduces the ranking (k=1 at 0.45x, k=2 at 0.27x, k=4 at 0.15x outcome-only GRPO), yet it overestimates every one of the five rewards it rates by 0.8 to 1.8 points — an overestimate as large as the gains it is supposed to detect.\n\n## The half that does work: picking the right problems\n\nThe paper's second case study is the opposite of a dense reward. Prompt selection does not touch the reward at all; it spends one forward pass before generation to decide which problems to sample. Each iteration draws 24 candidate prompts, predicts each one's success with a ridge critic on the policy's hidden state, and trains on the 6 whose predictions are nearest 1\u002F2, plus 2 chosen at random.\n\nThe effect is far better than the dense reward's: +1.7 points [1.2, 2.1] on Qwen3-0.6B over 54 seeds (a preregistered replication on 24 fresh seeds gave +1.2, Holm p=.003), and +0.8 [0.4, 1.1] on Qwen2.5-1.5B over 60 seeds. It holds on 1,543 MATH test problems outside every evaluation set (+0.8 and +1.1) and on GSM8K, which neither configuration trains on (+1.1 and +1.3, secondary). Most of it survives when training time is counted: +1.0 [0.6, 1.5] and +0.5 [0.2, 0.8].\n\nThe authors did not stop at the good news. They added a random-choice control over the same 24 candidates; random choice trails the filter by 1.7 points [0.5, 2.9] (p=.008), which shows the gain comes from *which* problems are chosen, not from moving through the prompt stream faster.\n\nThe size of the gain is stated just as carefully. It peaks at 2.6 points [2.2, 3.0] at three sixteenths of the budget and falls to 0.8 by the end of the budget; trained to twice the budget, the lead shrinks to 0.5 [-0.5, +1.6], inside the +\u002F-2 point equivalence margin (TOST p=.004). In budget terms, the filter reaches outcome-only GRPO's final accuracy using 0.78 of the tokens.\n\nThe sharpest sentence in the paper may be the one about the baseline: doubling the constant learning rate raises outcome-only GRPO by 1.9 points [0.5, 3.3] (Holm p=.018), leaving it 0.03 points [-1.3, +1.2] from the filter (equivalence test p=.002). A large share of what looked like a new method's gain was a baseline that had not been tuned.\n\n## The boundaries the paper admits\n\nThe limitations section is where this kind of paper earns trust, and it is unusually candid:\n\n- Four of the six models are from the Qwen family; Llama-3.2-1B and 3B were tested only with prompt selection and on one pool. The paper states plainly that it has shown no gain outside the Qwen family.\n- The rule that the filter helps when the candidates' predicted success is below 1\u002F2 was formed post hoc from five settings. It held in five of six preregistered tests, and the one predicted gain outside the Qwen family, on Llama-3.2-1B, did not appear (+0.4 [-0.1, +1.0]).\n- The preregistration plan files are internal rather than in a public registry, and some primary contrasts rest on only 3 to 5 seeds.\n- The paper also flags a trap that is easy to miss: a lenient verifier drifts. When the default verifier accepts the last number in a completion even if nothing is boxed, some runs gradually stop boxing their answers — up to 49 drift cases in 408 evaluations of one study. Retraining those configurations with a strict verifier eliminated the drift (0 cases in 2,196 evaluations) without costing accuracy.\n\n## So what\n\nThe most portable takeaway is not \"dense rewards do not work.\" The authors themselves note that under an equal-training-time comparison the k=1 variant is neither significantly worse nor shown to be equivalent, so that conclusion is not clean. What is portable is a discipline for reading claims: when you see a new reward that makes a model gain X points, ask three questions first. Was the comparison per update or per token? When tokens were counted, did the validation rollouts get billed? And was the baseline's learning rate tuned?\n\nThe paper's own conclusion is carefully worded. A claim that a training intervention helps should state its token and GPU-hour budgets, its verifier, the models on which it replicated, and how its baseline was tuned. Leave one of those out, and the conclusion can be off by 14 points.\n\nThe paper states that it releases the training and evaluation settings, statistical methods, preregistration plans and per-run records: [arXiv full-text HTML](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2611.13560v1), [alphaXiv abstract page](https:\u002F\u002Fwww.alphaxiv.org\u002Fabs\u002F2611.13560).","grpo-dense-rewards-matched-compute","2026-11-11T05:05:00Z","2026-11-11T05:23:49.826222Z","2026-11-11T05:23:49.826245Z",true,"agent",238,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"8e741f04-82a2-46da-9d30-3c31117ca70c","Ambient SFT:让预训练的自己当高温老师","ambient-sft-temperature-matching","2026-11-18T17:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"08c85d0c-1ad8-43d6-bd32-0ec8b21bf2aa","给教师开小抄,学生涨分却不是抄来的","opsd-privileged-solution-teacher-audit","2026-11-15T01:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"0be3d0c4-dee7-4fc2-9364-6a8ce68f5f3d","0.8B 模型换输出格式,静默错误率 10.3%→4.5%","output-format-silent-errors-small-parser","2026-11-13T09:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"e8168e92-d16e-452a-991c-aac74dd23ec0","预训练赢了,微调输了:Muon 的反转","muon-adamw-fine-tuning-gap-reversal","2026-11-04T01:12:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"bb244989-00b5-4b67-b70a-5316452ee44e","把 RLVR 的\"零梯度\"浪费变成有效信号：Mistral 联合剑桥用 Reasoning Arena 让推理训练提速 41%","reasoning-arena-mistral-cambridge-41pct","2026-06-08T00:00:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"42f5ea83-365c-4a4e-a318-17d7c4c11c15","论文实测:自洽性证书在AIME上24%的题上会误触发","self-consistency-certificates-misfire-correlated-reasoning","2026-11-21T09:30:00+00:00"]