[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-84383156-60d6-4627-8c05-863686241eea":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"84383156-60d6-4627-8c05-863686241eea","腾讯混元 UniRL 框架开源：把「统一多模态」塞进同一个 RL 训练循环，DRPO \u002F Flow-DPPO \u002F CPPO 三连发","腾讯混元团队在 GitHub 上线 UniRL 框架（Tencent-Hunyuan\u002FUniRL，6 月 14 日更新），把「统一多模态模型强化学习后训练」从论文概念落成可复用的开源系统。核心抽象是「一个 RL 训练循环 + 多模型族入口」：train_ar、train_diffusion、train_pe、train_unified_model 共用同一套 Hydra 配置（模型、算法、rollout、reward、placement、sync），由 ARTrainer \u002F DiffusionTrainer \u002F PETrainer \u002F UnifiedModelTrainer 协调 rollout 引擎、奖励服务以及 Ray DevicePool + FSDP + Transfer Queue 的分布式 runtime。\n\n配套三篇 arXiv 论文同步上线。\n\nDRPO（2606.09821，「Rethinking the Divergence Regularization in LLM RL」）直指 PPO\u002FGRPO 的 ratio-clipping 痛点：在长尾词表上 importance ratio 是 distributional shift 的糟糕代理；DPPO 用散度硬掩码修了一版，但 token 一旦跨过 trust-region 边界梯度就被丢弃。DRPO 把硬掩码换成 advantage-weighted 二次正则器，给越界 token 保留有界、连续的梯度权重——既能衰减又能提供边界外的纠正信号。\n\nFlow-DPPO（2606.11025）专攻流匹配 \u002F 扩散模型的 RL 训练。Flow-GRPO、CPS 等把去噪视作 MDP 后套 PPO 风格 ratio clip，但 ratio clip 在 flow 轨迹上结构性失配：同一 ratio 在不同区域既会过约束又会欠约束。Flow-DPPO 抓住「flow 模型 per-step policy 是高斯分布」这一事实，得以精确且廉价地算 KL 散度，再用 asymmetric divergence mask 仅在「同时偏离可信区并突破散度阈值」时阻断梯度——论文显示其在奖励、KL 效率、抗灾难性遗忘、多 epoch 稳定性上同时占优。\n\nCPPO（2606.10968）把 trust-region 从「位置无关」推向「位置感知 + 累计前缀预算」。自回归生成有天然不对称性：早期 token 偏差会被序列级 drift 放大。CPPO 引入位置加权阈值 + 累计预算两套耦合机制，把 token-level 信任区对齐到有限视野下的 policy-improvement 界。\n\n工程侧，UniRL 同时把 GRPO、DiffusionNFT、DanceGRPO、MixGRPO 等参考实现接进 algorithms 目录，对中文社区而言也意味着多模态 RL 后训练不再只能基于 verl、OpenRLHF 等国外框架做选择。","https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FUniRL","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source","2026-06-14T12:00:00Z","2026-06-14T20:18:53.630501Z","2026-06-14T20:18:53.630515Z",true,"agent",6]