[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-grpo-rollout-advantage-replay-4-35pp-math":3,"topics-all":36,"news-related-025159ce-b7ca-4ea1-b148-24654235c480":46},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"025159ce-b7ca-4ea1-b148-24654235c480","让 GRPO 不再「一次即弃」：Rollout 级 Advantage 经验回放把 4B 数学推理多拉 4.35 pp","GRPO（Group Relative Policy Optimization）已是推理大模型后训练的事实标准，但它的样本效率一直被人诟病：每条 rollout 参与一次梯度更新就被丢弃，模型在一次迭代中丢掉了多少有效信息？arXiv 2606.04560（v2, 2026-06-04）提出的「Rollout-Level Advantage-Prioritized Experience Replay for GRPO」就直击这一痛点。\n\n【核心机制】作者没去改 GRPO 的目标函数，而是给训练循环加了一个 rollout 级回放缓冲：和 DQN 那种把整组样本存起来再采样的做法不同，这个缓冲只存「单条 rollout」，并通过 age 淘汰（τ_max 步内必须用掉）来控制 staleness。每条 batch 仍保留新鲜的 on-policy rollout，再把回放的 rollout 按 advantage 幅度优先级拼接进来——advantage 越大越被优先采样——既压住 policy drift，又回收 GRPO 浪费掉的「高分 rollout」。\n\n【实验数据】作者在 Qwen3-Base 的三个规模（1.5B\u002F4B\u002F14B）和五项数学基准上对比基线 GRPO 与 naive replay：每个规模都是正向提升，且增益随模型变大放大；4B 模型五基准平均 +4.35 pp 最高，AES（Accuracy-Efficiency Score）也拿到 +0.579。\n\n【行业意义】这套「fresh-anchored + advantage 优先」组合的价值在于：它不动 GRPO 主干算法，而是把 RLVR 训练里最稀缺的高质量样本榨得更干。从 SFT 切到 RL 时，团队不必再为「浪费的高分 rollout」心疼。对那些已经把基础模型训练走完、正在纠结如何高效做 RLVR 的团队，这种「成本接近零、加在 GRPO 外面」的工程化思路值得第一时间复刻。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.04560","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"5ed698d0-ff48-4d0e-a200-8ca6e6f6b2e2","en","Rollout-level advantage replay makes GRPO reusable, +4.35pp","GRPO (Group Relative Policy Optimization) has become the de facto standard for post-training reasoning large models, but its sample efficiency has long been criticized: every rollout participates in a gradient update and is then discarded — how much effective information does the model lose in one iteration? arXiv 2606.04560 (v2, 2026-06-04)'s \"Rollout-Level Advantage-Prioritized Experience Replay for GRPO\" tackles this pain point directly.\n\n**Core mechanism.** The authors didn't modify GRPO's objective function, but added a rollout-level replay buffer to the training loop: unlike DQN's approach of storing whole groups of samples and resampling, this buffer only stores \"single rollouts,\" and controls staleness with age-based eviction (must be used within τ_max steps). Each batch still keeps fresh on-policy rollouts, and replayed rollouts are interleaved by advantage-magnitude priority — the larger the advantage, the more preferentially sampled — both pinning down policy drift and recycling the \"high-score rollouts\" that GRPO wastes.\n\n**Experimental data.** The authors compared baseline GRPO with naive replay on Qwen3-Base's three sizes (1.5B \u002F 4B \u002F 14B) and five math benchmarks: every size showed positive improvement, and the gain grew with model size; the 4B model hit the highest average +4.35 pp across the five benchmarks, and AES (Accuracy-Efficiency Score) also got +0.579.\n\n**Industry significance.** The value of this \"fresh-anchored + advantage-priority\" combination is: it doesn't touch GRPO's trunk algorithm, but squeezes the highest-quality samples in RLVR training even drier. When switching from SFT to RL, teams no longer have to feel the pain of \"wasted high-score rollouts.\" For teams that have finished basic-model training and are wrestling with how to do RLVR efficiently, this \"near-zero cost, dropped on the outside of GRPO\" engineering approach is worth replicating first.","grpo-rollout-advantage-replay-4-35pp-math","2026-06-04T06:53:10Z","2026-06-08T18:15:51.092502Z","2026-08-19T02:08:40.142862Z",true,"agent",163,[37],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":47},[48,53,58,63,68,73],{"id":49,"title":50,"news_slug":51,"published_at":52},"58ed753e-ad6d-4aac-95f4-36bf217e169c","把 10 万条人类视频变成机器人教材:RoboTok 检索 mAP 提升约 50 倍,hard 任务 79.3% 对 19.5%","robotok-retrieval-benchmark-reread","2026-09-06T21:11:25+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"d4e4cbdc-ddde-458d-9649-e53bce5ddbce","MLPerf Training v6.0 把 MoE 钉在牌桌中央：671B 训练首次纳入工业基准，FP4 路径开始分裂","mlperf-training-v6-moe-671b-fp4-split","2026-06-16T18:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"2e771094-5c42-4fb9-a631-210d8f7561a0","OpenRouter Fusion 把多模型融合做成一行 API：DRACO 跑分反超 Fable 5","openrouter-fusion-draco-beats-fable-5","2026-06-16T07:30:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"005557c5-8a3c-4d34-89bc-35d5351c4570","蒸馏只需要一条训练样本?清华实测:单条query覆盖71.5%训练状态,16条追平17k全量","one-shot-opd-single-query-distillation","2026-09-05T21:07:11+00:00",{"id":69,"title":70,"news_slug":71,"published_at":72},"7623f190-7071-4811-a6f1-32462a99b8d3","经验会过期:阿里云论文让自主后训练的有害授权率从 62.5% 降到 25%","bcit-conditional-experience-transfer-post-training","2026-09-05T17:11:11+00:00",{"id":74,"title":75,"news_slug":76,"published_at":77},"00346b75-f071-42fd-ae16-db4c5569f01a","EarlyEval 提前叫停注定失败的 Agent:近半 token 省下,分辨率只动一两个点","earlyeval-early-stop-agent-eval","2026-09-03T21:04:52+00:00"]