The dominant cost of RL training for large models is not the gradient update but the rollout: the model has to generate large volumes of long reasoning trajectories, then score them with verifiable rewards. As generation scales, KV cache eats the memory and bandwidth — and asynchronous rollout architectures amplify the problem.
The hidden bottleneck of async rollout: KV concentration
A KAIST team (authors Sihwan Park, Sung-Yub Kim, Doohyuk Jang, Eunho Yang) named this phenomenon in a paper submitted to arXiv on October 10: KV concentration. Asynchronous rollout keeps GPUs busy through sustained concurrency, cutting idle time during long-tail generation. But under sustained concurrency, context processing gets squeezed into fewer decode steps, and the KV workload per step balloons. In matched-workload comparisons, the team also finds that KV cache quantization delivers substantially greater acceleration than weight-only or weight+activation quantization — the bottleneck is in the KV, not the weights.
The most aggressive INT4 KV quantization is fastest, but the paper shows it degrades learning stability, even with token-level truncated importance sampling (TIS): the sampling distribution gets skewed, and the training signal skews with it.
INT4 as the base, INT8 insurance for critical positions
KV4-RL's answer is layered protection: the bulk of the context is stored with calibration-free INT4 KV quantization — described in the paper as a paged INT4 format — while an INT8 side cache protects a bounded set of critical positions: attention sinks and recent context, handled with higher-fidelity probability-value arithmetic. Bounding the protected positions reduces sampler mismatch while retaining most of the INT4 speedup.
1.8x speedup, under 0.5pp accuracy loss
The numbers all come from 400 iterations of Qwen3-8B-Base GRPO training: 1.80x cumulative rollout speedup, 1.27x full-iteration speedup, and 1.97x rollout speedup over the final 50 iterations. On the accuracy side, under a common BF16 evaluator, the six-benchmark mean accuracy of both Qwen3-4B-Base and Qwen3-8B-Base stays within 0.5 percentage points of full-BF16-rollout training. The paper runs 32 pages with 15 figures and 6 tables under a CC-BY 4.0 license, and the HF papers page already lists one companion Space, with community traction still climbing.
Inference-side KV work is crowded; training-side is just starting
The past month has been busy for this track — training KV caches as neural networks, tuning per-layer compression ratios, moving KV out of VRAM, entropy-coding it down to the 2-bit level; this site has covered them all, but every one tells an inference-side (serving) story. KV4-RL switches axes: the rollout phase of RL training is itself "generation as a service," and asynchronous scheduling changes the KV cost structure entirely. Training-side KV engineering has seen almost no systematic study; the paper's matched-workload comparison (KV quantization vs weight-only vs weight+activation) fills exactly that gap.
For teams running RLVR, this is a directly computable trade: the larger the share of rollout in total iteration time, the more a 1.8x rollout speedup pays off end-to-end — and "almost no accuracy loss" means it can be adopted without touching the training recipe. As RL post-training becomes the standard pipeline for large models, rollout-side memory and bandwidth engineering is likely the next corner to get crowded.
Refs: arXiv:2610.14448 (https://arxiv.org/abs/2610.14448) · HF papers (https://huggingface.co/papers/2610.14448)