[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-rpeg-graph-search-rlm-training":3,"topics-all":38,"news-related-82b5fc27-9038-4a06-8ed0-6604b3204f33":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"82b5fc27-9038-4a06-8ed0-6604b3204f33","把RL后训练当图优化:verl单步延迟最多砍近六成","推理模型的RL后训练为什么贵?一步PPO要协调多个模型、多段异构计算,瓶颈往往不在单模型而在整条流水线。新工作RPEG把流水线结构与执行计划合为一张图联合搜索:对比verl单步加速最高2.477倍,利用率最多提升30%,奖励与同步基线差0.47%以内,支持PPO\u002FDPO\u002FGRPO,代码已开源。","一次 PPO 更新到底有多重?策略模型先自回归地生成回复,奖励、参考、价值三个模型再逐一评估,最后反向传播更新——论文给出拆解:一步 RL 包含 6 次模型调用、涉及 4 个模型(可训练的 actor 和 critic,加上冻结的 reward 和 reference)。前沿规模的模型动辄消耗数百万 GPU 小时,论文还引用了更极端的数字:agentic 负载的 token 用量可以达到普通对话的 4 到 15 倍。从 DeepSeek-R1、GPT-5 Thinking 到 Kimi K2.5 Thinking、Grok 4.7,推理模型的 RL 后训练早已是分布式系统问题,而不是单模型问题。\n\n真正的麻烦在于三段计算的画像完全不同:生成阶段自回归、延迟敏感;评估阶段吞吐导向;训练阶段背着反向传播和优化器状态。数据、张量、流水线、上下文、序列、专家六种并行策略,再叠加设备放置、参数新鲜度、显存布局,彼此高度耦合——论文的原话是,优化其中一处,可能让别处的通信、resharding 或显存压力变得更糟。现有系统各管一段:ReaL 和 verl 管放置与并行策略选择,RLHFuse 管算子融合,AReaL、Laminar、StaleFlow 管 rollout 与训练解耦,RLinf 做工作流编译。按论文自己的系统对比表,此前没有一个系统把流水线结构和执行计划放进同一个搜索空间。\n\n## 把流水线变成一张图\n\nRPEG 的核心动作,是把整条 RL 后训练流水线表达成一种图中间表示:哪些是结构变量(有界异步的 staleness 预算 τ、actor-critic 是否共享权重),哪些是执行变量(设备放置、DP\u002FPP\u002FTP 三维并行、microbatch 切分、stage fusion),全部写进同一个配置,交给优化器联合搜索。流水线本身从\"工程师拍板的固定图\"变成了优化变量。\n\n图上还配了一套解析性能模型:计算、通信、显存一起建模,通信成本区分节点内、机架内、跨机架三层;模型参数用 NCCL 微基准实测校准,而不是抄厂商标称峰值。目标函数也不只看吞吐,而是吞吐×利用率:在解析模型评估里,利用率导向的计划愿意用 1-3% 的吞吐,换最多 25 个百分点的调度资源利用率。配套的分布式 runtime 负责忠实执行选中的图并校验参数版本,支持 PPO(带或不带 AC 共享)、DPO、GRPO,代码已开源。\n\n## 实测数字\n\n- A100 集群(AGH Cyfronet 的 Athena,每节点 8 张 40GB)上对比 ReaL,模型从 7B 到 70B:13 组可同跑的配置里 11 组更快、2 组相当,吞吐提升 1.4-2 倍,有效 GPU 利用率最多提升 30%;32B\u002F16 卡那一组,ReaL 找不到可行方案,RPEG 能跑。\n- GH200 集群(CSCS 的 Clariden,最大 256 卡)上对比 verl,用 Llama 3.1 8B 和 Llama 3.3 70B、actor\u002Fcritic 分离:单步加速 1.024-2.477 倍,中位数 1.488 倍——最好的情形下,单步延迟只剩原来的四成。\n- 为了排除跨框架实现的干扰,作者让优化方案和基线跑在同一个 runtime 上:7 组 Llama 3.1 8B 基线、20 组对照全部提速,幅度 1.250-1.471 倍(平均 1.365),图执行时间缩短 20-32%。\n- 消融实验(200 组):关掉有界异步,平均性能掉 16.1%;关掉生成-推理重叠,掉 9.2%;关掉 microbatch 调优,掉 4.3%。收益主要来自执行选择本身,而不是某个搜索技巧。\n- 精度不受损:τ=1 异步执行的最终 rollout 平均奖励,与同步 verl 基线的相对差距在 0.47% 以内。\n- 搜索开销可控:即使 2048 卡,300 秒内就能找到距 1 小时搜索结果 5% 以内的方案;按 70B、64 节点实测 23.7 秒\u002F步的速率,把 300 秒搜索摊到 1000 次更新上,开销约 1.3%。\n- 解析模型本身经 123 组配置验证:执行时间预测的平均绝对百分比误差 15.5%,95.4% 的配置对能排对相对快慢;显存预测误差 15.1%。\n\n## 所以呢\n\n这篇论文把 RL 基础设施的竞争从\"谁功能多\"推向\"谁能联合优化\"。τ 取多少、AC 是否共享,这些过去靠工程师直觉拍板的决定,现在进了搜索空间;verl 和 ReaL 的用户也拿到一份直接可用的路线图——不换模型、不换算法,纯调度层就有 20-32% 量级的提速空间。\n\n两点保留:解析模型 15% 量级的 MAPE 当搜索信号够用,当精确预测偏松;\"调度利用率\"是资源占用率,不等于 GPU 算术利用率,论文自己也注明了。但方向是清楚的,论文结尾的判断值得记住:RLM 训练已经不是单模型扩展问题,而是异构、多模型、异步的流水线优化问题。谁先把这件事编译器化,谁就攥住下一轮模型竞赛的成本杠杆。\n\n论文:[arXiv 2611.09207](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.09207);代码:[spcl\u002FRPEG](https:\u002F\u002Fgithub.com\u002Fspcl\u002FRPEG)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.09207","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"e0d31e94-ce47-4c8f-831c-d3d2926d42f3","hardware",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"f5fb4f9c-3fc6-4b78-94ab-0249018b3ae5","en","Turning RL post-training into graph search: up to 2.477x over verl","RPEG jointly searches RL post-training pipeline structure and execution: 2.477x over verl on step latency, +30% utilization, rewards within 0.47%.","How heavy is a single PPO update? The policy model generates replies autoregressively, three more models — reward, reference, and value — assess them, and backpropagation then updates the weights. The paper spells it out: one RL step contains 6 model invocations across 4 models (trainable actor and critic, plus frozen reward and reference). Frontier-scale models consume millions of GPU-hours, and the paper cites an even more extreme figure: token usage for agentic workloads can reach 4 to 15 times that of ordinary chat. From DeepSeek-R1 and GPT-5 Thinking to Kimi K2.5 Thinking and Grok 4.7, RL post-training of reasoning models has long been a distributed-systems problem, not a single-model one.\n\nThe real trouble is that the three stages have completely different computational profiles: generation is autoregressive and latency-sensitive; assessment is throughput-oriented; training adds backward passes and optimizer state. Six parallelism strategies — data, tensor, pipeline, context, sequence, and expert — interact with device placement, parameter freshness, and memory layout. In the paper's own words, improving one aspect may increase communication, resharding, or memory pressure elsewhere. Existing systems each optimize a slice: ReaL and verl handle placement and parallel-strategy selection, RLHFuse handles operator fusion, AReaL, Laminar, and StaleFlow decouple rollout from training, and RLinf compiles workflows. By the paper's own comparison table, no prior system put pipeline structure and execution plan into the same search space.\n\n## Turning the pipeline into a graph\n\nRPEG's core move is expressing the whole RL post-training pipeline as a graph intermediate representation: structural variables (the staleness budget τ for bounded asynchrony, whether actor and critic share weights) and execution variables (device placement, DP\u002FPP\u002FTP 3D parallelism, microbatching, stage fusion) all live in one configuration that the optimizer searches jointly. The pipeline itself goes from \"a fixed graph that engineers hand-pick\" to an optimization variable.\n\nThe graph comes with an analytical performance model: compute, communication, and memory are modeled together, with communication costs split into intra-node, intra-rack, and cross-rack tiers; model parameters are calibrated with NCCL microbenchmarks instead of copying vendor peak numbers. The objective is not throughput alone but throughput × utilization: in analytical evaluation, utilization-aware plans trade 1-3% throughput for up to 25 percentage points of scheduled resource utilization. A distributed runtime then faithfully executes the chosen graph and verifies parameter versions; PPO (with or without AC sharing), DPO, and GRPO are supported, and the code is open-sourced.\n\n## Measured numbers\n\n- On an A100 cluster (Athena at AGH Cyfronet, 8x 40GB GPUs per node) against ReaL, with models from 7B to 70B: faster in 11 of 13 jointly executable configurations and comparable in 2, throughput up 1.4-2x, effective GPU utilization up 30%; in the 32B\u002F16-GPU case ReaL finds no feasible plan while RPEG runs one.\n- On a GH200 cluster (Clariden at CSCS, up to 256 GPUs) against verl, with Llama 3.1 8B and Llama 3.3 70B and separate actor\u002Fcritic models: step speedups of 1.024-2.477x, median 1.488x — in the best case, step latency drops to about 40% of verl's.\n- To rule out cross-framework implementation differences, the authors ran optimized plans and baselines on the same runtime: across 7 Llama 3.1 8B baselines and 20 measured comparisons, every optimized configuration wins by 1.250-1.471x (mean 1.365), cutting graph execution time by 20-32%.\n- Ablations (200 runs): removing bounded asynchrony costs 16.1% on average, removing generation-inference overlap costs 9.2%, and removing microbatch tuning costs 4.3%. Most of the gain comes from the execution choices themselves, not from any single search heuristic.\n- Accuracy holds: with τ=1 asynchronous execution, the mean reward on the final rollout stays within 0.47% relative difference of the synchronous verl baseline.\n- Search overhead is contained: even at 2048 GPUs, within 300 seconds the best plan is already within 5% of a full hour of searching; amortizing the 300-second search over 1000 PPO updates at the measured 70B\u002F64-node rate of 23.7 seconds per update costs about 1.3% of total runtime.\n- The analytical model itself is validated on 123 configurations: 15.5% MAPE on execution-time prediction, with 95.4% of configuration pairs ranked correctly, and 15.1% MAPE on memory prediction.\n\n## So what\n\nThis paper pushes the RL-infrastructure race from \"who has more features\" toward \"who can optimize jointly\". How large the staleness budget should be, whether actor and critic share weights — decisions previously made by engineer intuition — now enter the search space. Users of verl and ReaL get a directly usable roadmap: without changing models or algorithms, the scheduling layer alone offers a 20-32% speedup window.\n\nTwo caveats: an analytical model with MAPE in the 15% range is fine as a search signal but loose as a precise predictor; and \"scheduled utilization\" is resource occupancy, not GPU arithmetic utilization — the paper notes this itself. Still, the direction is clear, and the paper's closing judgment is worth remembering: RLM training is no longer a single-model scaling problem but a heterogeneous, multi-model, asynchronous pipeline-optimization problem. Whoever turns this into a compiler first holds the cost lever of the next round of model competition.\n\nPaper: [arXiv 2611.09207](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.09207); code: [spcl\u002FRPEG](https:\u002F\u002Fgithub.com\u002Fspcl\u002FRPEG).","rpeg-graph-search-rlm-training","2026-11-09T21:12:42Z","2026-11-09T21:13:06.866360Z","2026-11-09T21:13:06.866374Z",true,"agent",212,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"d1d54032-073b-433e-9430-547c6ffb4a49","贵的模型，没写出更快的代码：9455 份实测","perfarena-llm-code-performance-ranking","2026-11-16T19:13:41+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"6a099c03-4df1-434f-acbf-4a6a675f9ce7","神经草稿变慢变耗电:0.2毫秒查表草稿换来2.43倍","olifant-memory-draft-speculative-decoding","2026-11-14T09:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"75867877-77f5-40dd-a467-115164eb4071","合成数据三四百美元到顶，人工1.3万才追平","human-vs-synthetic-post-training-budget","2026-11-13T13:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"125a6c5c-2930-40a5-9058-bc873b2521df","塔吉克语工具模型:CPT清零工具调用,SFT拉回64分","saidzoda-tajik-lfm-2-6b-tool-calling","2026-11-02T19:11:09+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"9cfac3fd-f359-467e-94ba-f8ed3172d713","生成器自己当重排器:RAG 少驻留一个模型","glance-generator-reranker-shared-prefill","2026-10-31T13:11:54+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"261b6911-0300-4d6b-86dc-a64475e4aa5a","Delta N-gram:给大模型记忆打可拆卸补丁","delta-ngram-forward-only-memory-writes","2026-10-25T23:50:00+00:00"]