The effect of RL training on LLMs highly depends on the logprobs (log probabilities) returned by the inference engine — any tiny difference will be amplified in gradient updates. But vLLM's V0 to V1 is an underlying rewrite, and migration is far from a simple "upgrade." ServiceNow team's PipelineRL documented this pitfall process.
When they switched their inference backend from vLLM 0.8.5 to 1.18.1, training dynamics completely collapsed — clip rate, KL divergence, entropy, and reward curves all deviated from the V0 baseline. The cause wasn't in the RL objective function, but in the inference backend itself.
The team identified four issues: the rollout logprobs computation path has different semantics in V1; V1 has new runtime defaults; the inflight weight-update path has differences; and lm_head output precision is insufficient (fp32 vs lower precision). After fixing each, V1's final trajectory almost perfectly reproduced V0.
This case reveals a counter-intuitive principle: the inference engine's correctness must take priority over RL objective tuning. A seemingly tiny difference in the inference engine, in the long sequences of RL training, will be continuously amplified, eventually leading to completely different convergence paths. As vLLM V1 becomes the mainstream inference engine, RL training frameworks need to seriously treat this kind of migration compatibility — it's not just changing a version number, but revalidating the entire training pipeline.
For teams exploring RL+LLM, this experience is worth remembering: before pursuing better RL objectives, first ensure the inference backend is mathematically equivalent.