[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-vllm-v0-v1-migration-servicenow-correctness":3,"news-related-93637a71-d655-4aae-a7e0-f0e9a6383228":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"93637a71-d655-4aae-a7e0-f0e9a6383228","vLLM V0 迁移 V1：强化学习训练为何要把推理正确性放在首位","强化学习（RL）训练 LLM 的效果高度依赖推理引擎返回的 logprobs（对数概率），任何细微差异都会在梯度更新中被放大。但 vLLM 从 V0 到 V1 是一次底层重写，迁移绝非简单「升级」。ServiceNow 团队的 PipelineRL 记录了这次踩坑过程。\n\n他们在将推理后端从 vLLM 0.8.5 切换到 1.18.1 时发现：训练动态完全崩溃——clip rate、KL 散度、熵和 reward 曲线全都偏离 V0 基线。原因不在 RL 目标函数，而在推理后端本身。\n\n团队定位了四个问题：rollout logprobs 的计算路径在 V1 中语义不同；V1 有新的运行时默认值；inflight weight-update 路径存在差异；以及 lm_head 输出精度不足（fp32 vs更低精度）。逐一修复后，V1 最终轨迹几乎完美复现 V0。\n\n这个案例揭示了一个反直觉的原则：**推理引擎的正确性必须优先于 RL 目标的调优**。推理引擎一个看似微小的差异，在 RL 训练的长序列中会被持续放大，最终导致完全不同的收敛路径。随着 vLLM V1 成为主流推理引擎，RL 训练框架需要严肃对待这类迁移兼容性问题——不是换版本号，而是重新验证整个训练管道。\n\n对于正在探索 RL+LLM 的团队，这条经验值得记取：在追求更优 RL 目标之前，先确保推理后端在数学上是等价的。","https:\u002F\u002Fhuggingface.co\u002Fblog\u002FServiceNow-AI\u002Fcorrectness-before-corrections","24d5c6c5-6573-4180-a1fd-f1459842d1af",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"3c481794-5edf-4f49-a9f8-6cdb302d82bb","en","vLLM V0 to V1: why RL training must put inference correctness first","The effect of RL training on LLMs highly depends on the logprobs (log probabilities) returned by the inference engine — any tiny difference will be amplified in gradient updates. But vLLM's V0 to V1 is an underlying rewrite, and migration is far from a simple \"upgrade.\" ServiceNow team's PipelineRL documented this pitfall process.\n\nWhen they switched their inference backend from vLLM 0.8.5 to 1.18.1, training dynamics completely collapsed — clip rate, KL divergence, entropy, and reward curves all deviated from the V0 baseline. The cause wasn't in the RL objective function, but in the inference backend itself.\n\nThe team identified four issues: the rollout logprobs computation path has different semantics in V1; V1 has new runtime defaults; the inflight weight-update path has differences; and lm_head output precision is insufficient (fp32 vs lower precision). After fixing each, V1's final trajectory almost perfectly reproduced V0.\n\nThis case reveals a counter-intuitive principle: **the inference engine's correctness must take priority over RL objective tuning.** A seemingly tiny difference in the inference engine, in the long sequences of RL training, will be continuously amplified, eventually leading to completely different convergence paths. As vLLM V1 becomes the mainstream inference engine, RL training frameworks need to seriously treat this kind of migration compatibility — it's not just changing a version number, but revalidating the entire training pipeline.\n\nFor teams exploring RL+LLM, this experience is worth remembering: before pursuing better RL objectives, first ensure the inference backend is mathematically equivalent.","vllm-v0-v1-migration-servicenow-correctness","2026-05-07T04:10:00Z","2026-05-07T04:06:26.420837Z","2026-08-19T02:08:40.142862Z",true,"agent",115,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"183fb3be-e062-47e7-9591-7c2372e116c1","LLM 蒸馏的显存瓶颈不只在教师模型：离线 Top-K 与分块 KL 把长上下文训练装回单卡","llm-distillation-offline-top-k-chunked-kl","2026-08-05T20:08:13+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"2a312a41-8284-4429-92a8-d68559800cbe","神经符号AI新突破：能耗降低100倍，机器人任务准确率显著提升","tufts-neuro-symbolic-ai-100x-energy-95pct","2026-05-18T16:01:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"f7eaae9c-a0b0-437e-a2f9-77c8fd1bf59e","vLLM 2026 Q2 RL路线图：推理引擎为何要成为RL训练的一等公民","vllm-2026-q2-rl-roadmap-fp4-rdma","2026-05-07T01:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"5da227db-f53d-4b07-a0c6-4ea16e04cd4d","CURE 用不确定性焦点做「block-parallel 投机解码」：端到端 2.66–3.49×、接受长度涨 4.2–7.5%","cure-block-parallel-speculative-decoding","2026-08-08T02:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00"]