PPO in large-model reinforcement learning commonly uses a critic to estimate state values and reduce the variance of policy updates. A joint team from Shanghai Jiao Tong University, Shanghai AI Laboratory and five other institutions recently gave this component a systematic checkup: they identify a previously unnamed systematic failure mode in the critic, and propose a fix whose change is surprisingly small. The paper, "Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening", is on arXiv (2609.18708) and reached #3 on Hugging Face Daily Papers.

Value Flattening: critics play flat

The team estimates the true value of every intermediate state in a reasoning response using multiple Monte Carlo (MC) continuations, then compares those with PPO critic predictions. MC values swing sharply within a response, while critic predictions stay close to a flat line. The team names this phenomenon Value Flattening. It reproduces in a controlled FrozenLake environment and grows more pronounced as the state space expands — which explains why it bites especially hard on LLM reasoning, where responses span thousands of tokens.

Two root causes

  • Implicit variance penalty. Under the common setting of terminal-only rewards with γ = λ = 1, every state in a response shares the same sampled terminal return. Decomposing the dense MSE loss yields a term that directly penalizes within-response prediction variance — for the critic, flattening predictions is the cheapest way to minimize the loss.
  • Redundant neighboring updates. Two adjacent LLM states differ by a single token and share almost their entire history, so their representations and gradients are highly similar. Dense token-level supervision repeats nearly the same update hundreds of times: no new information gets in, while the variance-penalty side effect accumulates.

SP³O: supervise three states per response

The fix, SParse Proximal Policy Optimization (SP³O), keeps the actor objective, rollout procedure and return targets untouched, and applies the critic loss only at a few well-separated states per response. The paper reports that supervising three states per response suffices. The ablation shows that anchors fixed at 30%/60%/90% response progress average 45.57%, clearly above random placement at 36.59% (dense PPO baseline: 37.60%); sparse configurations with 3-8 supervised states beat denser variants, and the denser the supervision, the closer results fall back toward the dense PPO baseline.

Benchmarks (self-reported)

On Qwen3-4B-Base and Qwen3-8B-Base (in-domain trained on DAPO-Math-17k; math benchmarks averaged over 32 generations): the 4B model averages 45.57% across seven math benchmarks, above GRPO's 39.26% and PPO's 37.60%, with AIME26 rising from PPO's 14.69% to 22.08%; the 8B model averages 50.51%, also first. On out-of-distribution suites (ARC-C, MMLU-Pro, GPQA and others, averaged over four generations), the averages are 59.28% and 66.37%, both the highest of the three methods. To be fair, it is not a clean sweep: on 8B Minerva, GRPO scores 52.06% versus SP³O's 47.40%; on the 8B out-of-distribution suite, ARC-C and MMLU-Pro are led by PPO (93.84%) and GRPO (66.27%) respectively. SP³O's edge is on the average.

So what

The code is built on THUDM/slime v0.2.4, open-sourced under Apache-2.0, with an eight-GPU script for the 4B run — for teams already training with PPO, this is a near-zero-migration critic-side change worth A/B testing directly. The more interesting part is the signal: GRPO already sits in the comparison table as the default reference, and this paper demonstrates the other path — not sidestepping the critic, but accepting that dense supervision is unfriendly to it, and treating supervision itself as a scarce resource to design around. The next RLHF gain may come not from a larger reward model, but from a finer dissection of the components we already have.

References: arXiv 2609.18708; github.com/Dodojordi/SP3O