How heavy is a single PPO update? The policy model generates replies autoregressively, three more models — reward, reference, and value — assess them, and backpropagation then updates the weights. The paper spells it out: one RL step contains 6 model invocations across 4 models (trainable actor and critic, plus frozen reward and reference). Frontier-scale models consume millions of GPU-hours, and the paper cites an even more extreme figure: token usage for agentic workloads can reach 4 to 15 times that of ordinary chat. From DeepSeek-R1 and GPT-5 Thinking to Kimi K2.5 Thinking and Grok 4.7, RL post-training of reasoning models has long been a distributed-systems problem, not a single-model one.
The real trouble is that the three stages have completely different computational profiles: generation is autoregressive and latency-sensitive; assessment is throughput-oriented; training adds backward passes and optimizer state. Six parallelism strategies — data, tensor, pipeline, context, sequence, and expert — interact with device placement, parameter freshness, and memory layout. In the paper's own words, improving one aspect may increase communication, resharding, or memory pressure elsewhere. Existing systems each optimize a slice: ReaL and verl handle placement and parallel-strategy selection, RLHFuse handles operator fusion, AReaL, Laminar, and StaleFlow decouple rollout from training, and RLinf compiles workflows. By the paper's own comparison table, no prior system put pipeline structure and execution plan into the same search space.
Turning the pipeline into a graph
RPEG's core move is expressing the whole RL post-training pipeline as a graph intermediate representation: structural variables (the staleness budget τ for bounded asynchrony, whether actor and critic share weights) and execution variables (device placement, DP/PP/TP 3D parallelism, microbatching, stage fusion) all live in one configuration that the optimizer searches jointly. The pipeline itself goes from "a fixed graph that engineers hand-pick" to an optimization variable.
The graph comes with an analytical performance model: compute, communication, and memory are modeled together, with communication costs split into intra-node, intra-rack, and cross-rack tiers; model parameters are calibrated with NCCL microbenchmarks instead of copying vendor peak numbers. The objective is not throughput alone but throughput × utilization: in analytical evaluation, utilization-aware plans trade 1-3% throughput for up to 25 percentage points of scheduled resource utilization. A distributed runtime then faithfully executes the chosen graph and verifies parameter versions; PPO (with or without AC sharing), DPO, and GRPO are supported, and the code is open-sourced.
Measured numbers
- On an A100 cluster (Athena at AGH Cyfronet, 8x 40GB GPUs per node) against ReaL, with models from 7B to 70B: faster in 11 of 13 jointly executable configurations and comparable in 2, throughput up 1.4-2x, effective GPU utilization up 30%; in the 32B/16-GPU case ReaL finds no feasible plan while RPEG runs one.
- On a GH200 cluster (Clariden at CSCS, up to 256 GPUs) against verl, with Llama 3.1 8B and Llama 3.3 70B and separate actor/critic models: step speedups of 1.024-2.477x, median 1.488x — in the best case, step latency drops to about 40% of verl's.
- To rule out cross-framework implementation differences, the authors ran optimized plans and baselines on the same runtime: across 7 Llama 3.1 8B baselines and 20 measured comparisons, every optimized configuration wins by 1.250-1.471x (mean 1.365), cutting graph execution time by 20-32%.
- Ablations (200 runs): removing bounded asynchrony costs 16.1% on average, removing generation-inference overlap costs 9.2%, and removing microbatch tuning costs 4.3%. Most of the gain comes from the execution choices themselves, not from any single search heuristic.
- Accuracy holds: with τ=1 asynchronous execution, the mean reward on the final rollout stays within 0.47% relative difference of the synchronous verl baseline.
- Search overhead is contained: even at 2048 GPUs, within 300 seconds the best plan is already within 5% of a full hour of searching; amortizing the 300-second search over 1000 PPO updates at the measured 70B/64-node rate of 23.7 seconds per update costs about 1.3% of total runtime.
- The analytical model itself is validated on 123 configurations: 15.5% MAPE on execution-time prediction, with 95.4% of configuration pairs ranked correctly, and 15.1% MAPE on memory prediction.
So what
This paper pushes the RL-infrastructure race from "who has more features" toward "who can optimize jointly". How large the staleness budget should be, whether actor and critic share weights — decisions previously made by engineer intuition — now enter the search space. Users of verl and ReaL get a directly usable roadmap: without changing models or algorithms, the scheduling layer alone offers a 20-32% speedup window.
Two caveats: an analytical model with MAPE in the 15% range is fine as a search signal but loose as a precise predictor; and "scheduled utilization" is resource occupancy, not GPU arithmetic utilization — the paper notes this itself. Still, the direction is clear, and the paper's closing judgment is worth remembering: RLM training is no longer a single-model scaling problem but a heterogeneous, multi-model, asynchronous pipeline-optimization problem. Whoever turns this into a compiler first holds the cost lever of the next round of model competition.
Paper: arXiv 2611.09207; code: spcl/RPEG.