GRPO (Group Relative Policy Optimization) has become the de facto standard for post-training reasoning large models, but its sample efficiency has long been criticized: every rollout participates in a gradient update and is then discarded — how much effective information does the model lose in one iteration? arXiv 2606.04560 (v2, 2026-06-04)'s "Rollout-Level Advantage-Prioritized Experience Replay for GRPO" tackles this pain point directly.
Core mechanism. The authors didn't modify GRPO's objective function, but added a rollout-level replay buffer to the training loop: unlike DQN's approach of storing whole groups of samples and resampling, this buffer only stores "single rollouts," and controls staleness with age-based eviction (must be used within τ_max steps). Each batch still keeps fresh on-policy rollouts, and replayed rollouts are interleaved by advantage-magnitude priority — the larger the advantage, the more preferentially sampled — both pinning down policy drift and recycling the "high-score rollouts" that GRPO wastes.
Experimental data. The authors compared baseline GRPO with naive replay on Qwen3-Base's three sizes (1.5B / 4B / 14B) and five math benchmarks: every size showed positive improvement, and the gain grew with model size; the 4B model hit the highest average +4.35 pp across the five benchmarks, and AES (Accuracy-Efficiency Score) also got +0.579.
Industry significance. The value of this "fresh-anchored + advantage-priority" combination is: it doesn't touch GRPO's trunk algorithm, but squeezes the highest-quality samples in RLVR training even drier. When switching from SFT to RL, teams no longer have to feel the pain of "wasted high-score rollouts." For teams that have finished basic-model training and are wrestling with how to do RLVR efficiently, this "near-zero cost, dropped on the outside of GRPO" engineering approach is worth replicating first.