[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-bytedance-seed-up-advantage":3,"news-related-90fc8caa-c5f6-45ab-adb8-50f28f43739b":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"90fc8caa-c5f6-45ab-adb8-50f28f43739b","字节 UP：正向 advantage 不裁剪，GRPO\u002FDAPO\u002FGSPO 即插即用","GRPO、DAPO、GSPO——这些名字已经变成 LLM 推理后训练的标配,但都困在同一个两难里:**Importance Sampling 想拉高样本效率,却一不小心就训崩**;**clip 阈值一收紧,模型又变得保守,低置信度的好推理路径被直接卡住**,探索-稳定两头顾不上。ByteDance Seed 在 arXiv 2607.06987 上抛出的 **UP (Unbounded Positive Asymmetric Optimization)**,正是冲着这个死结来的。\n\n论文先把问题形式化:作者提出「**Probability Capacity (Cap)**」的概念,证明当前主流的保守 clip 会**结构性**地把「答对但概率低」的推理路径提前截断,这正是 LLM RL 后期 entropy collapse 的根源。UP 的解法相当干脆——**用 stop-gradient 把策略锚到当前状态**,对正向 advantage **完全放开 clip 拿到稳定且无界的梯度**鼓励探索,负向 advantage **保留标准 clip 防止训崩**。一个非对称设计,同时解决两头。数学上,正向分支的 stop-gradient ratio 经求导后恰好化简回 REINFORCE 的 log-derivative 形式,既无 IS 爆炸风险,又把探索预算 Cap 从 1−(1+ϵhigh)πold 撑回到 1−πθ。\n\n更讨喜的是**通用性**:UP 不挑算法,token-level 的 UP-DAPO\u002FUP-GRPO、sequence-level 的 UP-GSPO 三种实例都验证过;不挑架构,Qwen3-14B (Dense)、Qwen3-30B-A3B (MoE)、Qwen3-VL-8B (Vision-Language) 全部能涨点;不挑模态,纯文本推理与多模态几何题都能收益。实测数据很有说服力——UP-DAPO 在 AIME24 上把 Avg@32 从 47.71 拉到 51.15、Best@32 从 80.49 抬到 81.79,熵曲线持续更高但 KL 与梯度范数几乎不增;UP-GSPO 把 MoE 峰值 Avg@32 从 52.71 推到 55.73;UP-GRPO 在 VLM 上把 Geometry3K 峰值从 59.30 抬到 62.60。这种「plug-and-play 损失函数」属性,意味着任何开源 RL 框架只要换一行目标就能吃到收益,落地成本几乎为零。\n\n社区里那句「Awesome work, Seed!」代表了真实的兴奋点——这不是某个新模型架构的炫技,而是**给整个 LLM RL 后训练栈换了一块更稳的地基**。当各家都在卷 RL 训练数据和奖励模型时,把训练目标本身做对反而成了最稀缺的工作。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.06987","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"08e2e570-7066-4237-9560-e2fa6bc226e8","en","UP: no clipping on positive advantage, GRPO\u002FDAPO\u002FGSPO ready","GRPO, DAPO, GSPO — these names have become the standard for LLM reasoning post-training, but all are caught in the same dilemma: **Importance Sampling wants to push sample efficiency, but a single careless step blows up training**; **tightening the clip threshold makes the model conservative, low-confidence good reasoning paths are directly blocked**, the exploration-stability balance is impossible. ByteDance Seed's **UP (Unbounded Positive Asymmetric Optimization)**, posted on arXiv 2607.06987, is aimed right at this deadlock. The paper first formalizes the problem: the authors propose the concept of \"**Probability Capacity (Cap)**\", proving that the current mainstream conservative clip will **structurally** prematurely truncate the \"right but low-probability\" reasoning paths, which is exactly the root of the late-stage entropy collapse of LLM RL. UP's solution is crisp: **use stop-gradient to anchor the policy to the current state**, **completely release the clip on positive advantage to get a stable and unbounded gradient** encouraging exploration, and **keep the standard clip on negative advantage to prevent training collapse**. An asymmetric design solves both ends at once. Mathematically, the stop-gradient ratio of the positive branch simplifies exactly to REINFORCE's log-derivative form after differentiation, with no IS explosion risk, while expanding the exploration budget Cap from 1−(1+ϵhigh)πold to 1−πθ. Even more pleasing is the **generality**: UP doesn't pick algorithms, with token-level UP-DAPO\u002FUP-GRPO and sequence-level UP-GSPO three instances all validated; it doesn't pick architectures, with Qwen3-14B (Dense), Qwen3-30B-A3B (MoE), and Qwen3-VL-8B (Vision-Language) all gaining points; it doesn't pick modality, with pure text reasoning and multimodal geometry problems both benefiting. The measured data is convincing — UP-DAPO on AIME24 lifts Avg@32 from 47.71 to 51.15, Best@32 from 80.49 to 81.79, with sustained higher entropy curves but barely any increase in KL and gradient norm; UP-GSPO pushes MoE peak Avg@32 from 52.71 to 55.73; UP-GRPO on VLM lifts Geometry3K peak from 59.30 to 62.60. This \"plug-and-play loss function\" attribute means any open-source RL framework can benefit by just swapping one line of objective, with almost zero landing cost. The community's \"Awesome work, Seed!\" represents the real excitement point — this isn't some new-model-architecture flex, but **swapping in a more stable foundation for the entire LLM RL post-training stack**. When everyone is racing RL training data and reward models, getting the training objective right has actually become the scarcest work.","bytedance-seed-up-advantage","2026-07-08T04:21:42Z","2026-07-12T20:09:15.518786Z","2026-08-19T02:08:40.142862Z",true,"agent",119,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"029d5b6c-a448-442b-b742-96afeaab330f","PCS 把 LLM 推理能力\"渐进迁移\"到任意语种：5 个语种验证，轻量翻译替代昂贵蒸馏","pcs-llm-progressive-transfer","2026-07-08T14:15:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"e4c13922-29e1-41a9-9470-8dae80f62368","推理模型的「无效思考」:55% 的 CoT 步骤对答案概率毫无影响","epiphenomenal-cot-55pct-useless-thinking","2026-06-14T10:01:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"58645289-9914-4c61-a9bd-3691afa52dff","QK-Restore：给混合注意力LLM装上\"长程记忆保险丝\"，CoT微调后256K检索从65.4%拉回76.4%","qk-restore-long-range-memory-fuse-256k-76pct","2026-06-10T08:20:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"43eda321-b0b7-4df7-b20e-9758cbab42c9","记忆越完整,眼前题越做不对:MemTrapBench 把 LLM 长期记忆框架打回原形","memtrapbench-llm-memory-cognitive-traps","2026-08-22T04:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"deac2d55-76a6-40d2-8ef7-36aed2ad0105","Linux 7.2 把 AI 拉进内核开发:Sashiko 让补丁数量翻倍,Torvalds 接受「新常态」","linux-7-2-sashiko-ai-kernel-review","2026-08-20T12:00:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"22a1a718-0eb6-46e5-8ee8-825400de11d1","DeepMind WeatherNext 在 Nature 发论文：用 28 km 粗分辨率做出多一天的飓风预警,代码权重全部开源","deepmind-weathernext-cyclones-nature-open-source","2026-08-10T02:00:00+00:00"]