[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-90fc8caa-c5f6-45ab-adb8-50f28f43739b":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"90fc8caa-c5f6-45ab-adb8-50f28f43739b","UP:ByteDance Seed 用「正向 advantage 不裁剪」拆解 LLM RL 探索-稳定死结,GRPO\u002FDAPO\u002FGSPO 全部即插即用","GRPO、DAPO、GSPO——这些名字已经变成 LLM 推理后训练的标配,但都困在同一个两难里:**Importance Sampling 想拉高样本效率,却一不小心就训崩**;**clip 阈值一收紧,模型又变得保守,低置信度的好推理路径被直接卡住**,探索-稳定两头顾不上。ByteDance Seed 在 arXiv 2607.06987 上抛出的 **UP (Unbounded Positive Asymmetric Optimization)**,正是冲着这个死结来的。\n\n论文先把问题形式化:作者提出「**Probability Capacity (Cap)**」的概念,证明当前主流的保守 clip 会**结构性**地把「答对但概率低」的推理路径提前截断,这正是 LLM RL 后期 entropy collapse 的根源。UP 的解法相当干脆——**用 stop-gradient 把策略锚到当前状态**,对正向 advantage **完全放开 clip 拿到稳定且无界的梯度**鼓励探索,负向 advantage **保留标准 clip 防止训崩**。一个非对称设计,同时解决两头。数学上,正向分支的 stop-gradient ratio 经求导后恰好化简回 REINFORCE 的 log-derivative 形式,既无 IS 爆炸风险,又把探索预算 Cap 从 1−(1+ϵhigh)πold 撑回到 1−πθ。\n\n更讨喜的是**通用性**:UP 不挑算法,token-level 的 UP-DAPO\u002FUP-GRPO、sequence-level 的 UP-GSPO 三种实例都验证过;不挑架构,Qwen3-14B (Dense)、Qwen3-30B-A3B (MoE)、Qwen3-VL-8B (Vision-Language) 全部能涨点;不挑模态,纯文本推理与多模态几何题都能收益。实测数据很有说服力——UP-DAPO 在 AIME24 上把 Avg@32 从 47.71 拉到 51.15、Best@32 从 80.49 抬到 81.79,熵曲线持续更高但 KL 与梯度范数几乎不增;UP-GSPO 把 MoE 峰值 Avg@32 从 52.71 推到 55.73;UP-GRPO 在 VLM 上把 Geometry3K 峰值从 59.30 抬到 62.60。这种「plug-and-play 损失函数」属性,意味着任何开源 RL 框架只要换一行目标就能吃到收益,落地成本几乎为零。\n\n社区里那句「Awesome work, Seed!」代表了真实的兴奋点——这不是某个新模型架构的炫技,而是**给整个 LLM RL 后训练栈换了一块更稳的地基**。当各家都在卷 RL 训练数据和奖励模型时,把训练目标本身做对反而成了最稀缺的工作。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.06987","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm","2026-07-08T04:21:42Z","2026-07-12T20:09:15.518786Z","2026-07-12T20:09:15.518794Z",true,"agent",4]