Rescoring your own freshly generated reasoning trajectories while holding the reference answer—privileged on-policy self-distillation supplies exactly the dense token-level supervision that pure reinforcement learning lacks. But it has a known failure mode: the teacher's score and the verifier's verdict are two different signals. A trajectory that reads coherently yet lands on a wrong answer can still collect high teacher gains, and distillation keeps reinforcing that confidence. On September 3, a team from Tencent's HY LLM Frontier group and the University of Pennsylvania posted FlowBalance to arXiv, with a blunt fix: let the verifier decide direction, and let the self-teacher only refine magnitude.
Mechanism: sign gating plus trajectory energy
FlowBalance builds on FlowRL's distributional view. Each on-policy trajectory is first rescored by a frozen copy of the policy given privileged context such as a reference solution; token-level log-probability gains are clipped and averaged into a trajectory-level guidance score. That score is then combined with the verifier-derived group-relative advantage into a single trajectory energy, with the teacher term multiplied by the sign of the advantage. The gating logic is three lines: on verified positive-advantage trajectories, teacher support is retained; on rejected trajectories, teacher pressure is reversed; when the rollout group shows no outcome preference, the teacher term is disabled entirely. Profiled trajectory balance then fits the energy-reweighted distribution over complete responses, with one stopped log-partition estimate per rollout group and no separate token-level imitation loss. The paper also establishes within-group contrast preservation and a minimum-change reverse-KL characterization.
Numbers: five-benchmark averages and one glaring contrast
Across five mathematical reasoning benchmarks (step-180, five seeds), FlowBalance averages 64.26 on Qwen3-4B, +1.95 over GRPO, and 67.61 on Qwen3-8B, +2.12, taking the best mean on all five sub-benchmarks at the 8B scale with 89.33 AIME24 Pass@16. The 4B picture is not a clean sweep: FlowRL leads Minerva there, 51.99 versus 50.51. The most glaring contrast is direct OPSD: down 8.19 points on average at 4B, and a collapse to 41.16 at 8B—down 24.33 from GRPO—accompanied by shrinking response length, quantifying how privileged imitation crushes long reasoning chains. On training dynamics, FlowBalance reaches 0.5 AIME24 validation accuracy in roughly 100 steps versus about 143 for GRPO (a 1.43x speedup) and stays stable through 400 steps, while GRPO degrades sharply after around step 180. Ablations put the peak configuration at verifier coefficient 15 and teacher coefficient 1 (67.61 five-benchmark average).
Diversity: not just correct, but correct in more ways
The team used GPT-5.5 to extract the mathematical representation and tools from full AIME24 trajectories, then clustered anonymized summaries by semantic strategy and measured Simpson diversity over correct trajectories only: 0.2194 for FlowBalance, about 2.16x GRPO's 0.1017 and above RLSD's 0.1456. In one showcased tetrahedron problem, the common GRPO route runs through the Cayley–Menger determinant, while a FlowBalance trajectory recognizes 41=4²+5², 80=4²+8², and 89=5²+8², embeds the vertices in a 4×5×8 box, and computes the volume with a scalar triple product—both routes arrive at answer 104. The authors are explicit that this is a controlled, one-seed, LLM-judged diagnostic, not a population-level guarantee.
Caveats
Every number is self-reported; no third-party replication has appeared yet. Experiments cover mathematical reasoning on two Qwen3 scales only, and the code repository just landed (33 commits) with little community pickup so far. Porting the recipe to domains without clean verifiers—code, agents—carries unbudgeted cost. Still, the principle of "verifier sets the direction, dense signals only refine" gives every team mixing RLVR with self-distillation a clear rule: settle correctness first, then talk about scores.
References: arXiv:2609.03241 (https://arxiv.org/abs/2609.03241); project page alexhuang13.github.io/FlowBalance-Blog; code github.com/alexhuang13/FlowBalance