On-policy reinforcement learning for large language models may be quietly off-policy. Mainstream training stacks run two systems on the same weights: an inference engine samples the responses and a trainer computes the gradients — the configuration cited in the paper is vLLM sampling plus a separate trainer. When the numerics and weight versions of the two systems disagree, the assumption that training data comes from the current policy silently breaks. Earlier work already reported that vLLM and FSDP assign different token probabilities at identical weights. This paper, posted to arXiv on October 29, cuts the training-inference mismatch down to measurable components.

Two sources, two dynamics

The authors first subtract the infrastructure noise: top-p sampling makes the reported distribution differ from the one actually sampled, decoding and re-tokenizing changes the token sequence, and filtering responses by reward makes batch composition depend on the outcome. Once aligned, only two mismatch sources reach the algorithm layer: engine numerics and weight version. The split itself is clean — evaluate the trainer at the sampler's weights, and each sampled-token error divides exactly into a numerical discrepancy of the engine and a policy lag of the trainer. Their dynamics differ completely: the numerical discrepancy stays approximately constant through training, like a fixed bias that per-prefix score centering removes; the policy lag is produced by the trainer's own updates — the bias one update leaves enters the next update and enlarges the next lag, closing a positive feedback loop.

Experimental setup

Training runs in verl with vLLM samplers: Qwen3-0.6B and Qwen3-1.7B in non-thinking mode, the Countdown task, rewards of ±1, 64 prompts with 8 samples per step, SGD, 40 steps, three seeds per configuration. Four samplers: a BF16 reference with CUDA graphs (the on-policy baseline), an FP8 sampler (128x128 blocks) with dynamic activation quantization and an FP8 KV cache, a stale sampler that synchronizes only every 64 steps (so within 40 steps it stays at the initial weights), and their combination.

A failure law

Importance weighting cuts the loop only while the lag ratio stays concentrated. The paper's criterion is the effective sample size of the ratio: once it falls below one half, the correction fails. On Qwen3-1.7B the heavy tail brings the crossing to step 11, three steps before the Gaussian prediction, and importance sampling leaves the noise floor there. The magnitudes are stark: under quantization the corrections stay near the floor while the policy gradient reaches 13 times it, and score centering matches the 46.5% test accuracy of on-policy training; under staleness score centering reaches 32 times the floor and ends at 12.2% accuracy, importance sampling with centering stays at 2.1 times with 32.0%, and PPO referencing the trainer before each batch reaches 699 times.

DLC and the invisible collapse

The derived fix is the discrepancy-lag correction: center the numerical discrepancy, clip the lag ratio against the sampler's weights, and resynchronize the sampler when the effective sample size falls below one half. Under a stale sampler, DLC keeps deviation from on-policy training within twice the noise floor (1.9 times on Qwen3-0.6B); on the 1.7B model each of the three runs resynchronizes once, at steps 20, 35 and 36, and stays at the floor throughout. Adding the same-threshold resynchronization to importance sampling lowers its deviation from 115 to 2.6 times the floor. The most striking finding is how invisible the collapses are: without resynchronization, the off-support share — tokens where the initial policy's probability is below 10^-3 — reaches about 90% at the two collapse points, against at most 14% before them; with resynchronization it stays at or below 11% through step 40. Yet at the same time, the next-token KL on the sampler's prefixes remains within the range of healthy runs — the shift starts on prefixes the stale sampler never reaches, so monitoring only the sampler side hides the collapse. Across checkpoints the off-support share ranks the accuracy drop at a Spearman correlation of 0.80, above the 0.74 of the deviation itself.

So what

The value of the paper is not another training trick but turning an engineering pain point into a measurable object: constant numerics versus self-amplifying lag demand different fixes; the effective sample size falling below one half is the expiry date of importance weighting, and resynchronization is more fundamental than tuning coefficients. If you run asynchronous RL, add one item to the monitoring checklist — the off-support share of the trainer's own responses. Measure KL only on the sampler side, and you may stay in the dark until the run diverges.