In multi-turn household environments such as ALFWorld, training an agent means dealing with sparse episode-level reward: the policy may take a dozen steps before earning back a single 0 or 1. The paper opens with a concrete pair. On one task, the same checkpoint samples two trajectories, one a 14-turn success with 1,663 response tokens, the other a 50-turn failure with 6,446 tokens. Under the paper's accounting, that failure ends up representing 78.1% of the row-expanded outcome population and 79.5% of the token-weighted objective mass. The paper is Outcome-Mass Alignment (arXiv:2611.16816, https://arxiv.org/abs/2611.16816).
One trajectory is not one unit inside the trainer
There are two ways to hand a multi-turn interaction to the optimizer. Trajectory-wise optimization (RAGEN) concatenates the whole interaction into one sequence, so the effective context grows with the horizon. Stepwise optimization instead materializes each turn as a separate response row conditioned on a managed history, which keeps every individual context tractable but expands one source trajectory into a variable number of rows and tokens. GiGPO and HGPO follow the stepwise route, and so does this paper's analysis.
That expansion creates two problems at different levels. The first lives in the outcome statistics. The same episode outcome is attached to every row the trajectory produced, so row-based group statistics count it n_i times. The paper gives a minimal example: one successful trajectory (y=1, n=1) and one failed trajectory (y=0, n=3). Averaged over source trajectories the group mean is 0.5; averaged over expanded rows it becomes 0.25. The in-group baseline has been dragged down by how many rows the failure happened to expand into.
The second problem lives in objective mass. Under a token-mean loss, a trajectory receives a coefficient of D_i over the sum of D across the group, where D_i is its valid-token count. Under a row-mean loss it receives n_i over the sum of n. In other words, a failed trajectory, purely because it ran longer, naturally occupies more of the gradient.
OMA changes exactly two things
The method is called Outcome-Mass Alignment, and its two components can be enabled separately or together. Outcome Alignment computes the outcome statistics over unique source trajectories and then broadcasts the resulting advantage back to that trajectory's rows, so duplicating rows can no longer shift the statistic. Mass Alignment first aggregates token contributions within each trajectory and then gives every source trajectory the same outer mass, 1/K for a complete group, regardless of how many rows or tokens it expanded into.
What OMA deliberately leaves alone matters as much as what it changes: within-trajectory credit assignment, importance ratios, and clipping all stay native to the base optimizer. It only turns the question of how much influence one trajectory should carry from an implicit by-product into an explicit choice. GRPO, GiGPO, and GSPO can all host it.
Where it helps, and where it stops helping
The ALFWorld baseline is GRPO on Qwen2.5-1.5B/3B-Instruct, using the first 150 trainer updates, eight trajectories per task group, batch size 16, one PPO epoch, seed 0, and four GPUs. OMA lifts 1.5B success from 73.4% to 82.8% and 3B from 74.2% to 81.2%; the abstract phrases this as a 7.0 to 9.4 percentage-point gain in matched-horizon endpoint success.
The ablations make the point sharper, because the two components are not interchangeable. At 3B the baseline sits at 74.2%, Outcome Alignment alone actually drops to 61.7%, Mass Alignment alone reaches 77.3%, and only the combination gets to 81.2%. At 1.5B the split is more extreme: OA on its own yields just 38.3%, a long way below the 73.4% baseline, while MA alone gives 76.6% and full OMA gives 82.8%.
The boundary shows up on WebShop. On ordinary WebShop at 1.5B, endpoint success improves from 65.6% to 70.3% and task score from 76.2% to 78.8%, but success nAUC only moves from 39.1% to 39.0% and score nAUC falls from 63.3% to 57.9%. At 3B it reverses outright: success drops from 64.8% to 57.0% and score from 81.4% to 71.4%. The paper's explanation is that ordinary WebShop has weak or even reversed coupling between long trajectories and failure, so changing the accounting buys nothing. In a success-oriented variant, WebShop-SOL, that coupling is restored: success nAUC rises from 39.9% to 51.1% and endpoint success from 64.8% to 73.4%.
The result also survives a different optimizer. GiGPO goes from 88.3% to 89.8% endpoint success with nAUC from 63.1% to 67.3%; GSPO goes from 65.6% to 70.3% with nAUC from 42.8% to 49.2%.
The two corrections pull in opposite directions
The paper inspects this through a frozen-batch gradient audit, recomputing the gradients of the same batch under four accounting regimes. In the near-symmetric WebShop batch the cosine between OA and MA is 0.996. On ALFWorld, as expansion asymmetry grows, that value falls to -0.421, -0.506, and -0.368, with the GiGPO row at -0.667. Negative values mean the two corrections act in opposing directions rather than as a scalar rescaling. In the batch with the strongest asymmetry, where the failed-to-successful token ratio is 8.78, OMA pulls the gradient norm ratio down from 5.05 to 1.60, close to the raw regime.
Another team hit the same mechanism independently
Systematically longer failure trajectories are not something only this paper noticed. AsyncWebRL (arXiv:2606.05597, https://arxiv.org/abs/2606.05597) independently diagnoses, for multi-step GRPO on visual web agents, that the per-trajectory normalizer 1/|tau_i| down-weights the negative gradient on failed tokens, so the policy keeps producing verbose output structures; the authors break that coupling by replacing 1/|tau_i| with the constant 1/k. OMA names AsyncWebRL as its closest comparison on trajectory-length weighting while noting that the two choose different outer-weighting policies: OMA neither prefers shorter trajectories nor scales mass with H_i.
GiGPO (arXiv:2505.10978, https://arxiv.org/abs/2505.10978) takes a third route. Instead of touching outer accounting it builds anchor state groups for fine-grained credit assignment inside a trajectory, and the paper reports gains of more than 12% on ALFWorld and more than 9% on WebShop over GRPO. Taken together the three lines of work point at a shift in how multi-turn agent RL is being done, from tuning the optimizer alone to tuning the outer accounting as well.
The boundaries the paper draws itself
A few caveats need to travel with the numbers. The main-text experiments use a single seed, two environments, and only the 1.5B and 3B Qwen2.5 sizes. In the appendix two-seed runs, ALFWorld-3B mean endpoint success rises from 70.31% to 86.33%, but WebShop-1.5B nAUC moves the other way, with success nAUC from 30.29% to 29.15% and score nAUC from 45.37% to 43.89%; the paper calls these descriptive and reports no significance test. Matching trainer updates does not equalize generated tokens, environment interactions, or wall-clock time, so the comparison cannot be read as an efficiency result. The paper also states plainly that the WebShop versus WebShop-SOL contrast changes model scale and termination at the same time, so it is not an isolated causal test.
So what
The value of this paper is not the few percentage points. It is that it takes an accounting decision that has always been made silently and takes it apart for you. In multi-turn RL, how many samples a trajectory counts as in the optimizer's eyes was never the optimizer's decision; it was a side effect of how the training pipeline materialized the trajectory. Change the materialization and the weight on failure trajectories changes with it, and failed trajectories are usually the long ones.
So the next time a multi-turn agent training recipe reports a beautiful curve, it is worth asking two more questions: does it compute group statistics over rows or over source trajectories, and is its outer mass spread by token or by trajectory? The answers to those two questions may matter more than another sweep of hyperparameters.