Multi-turn agentic reinforcement learning has become close to the default recipe for training LLM agents, yet evaluation is still overwhelmingly trajectory-level: did the task get finished. The authors of this paper argue that this habit collapses two very different things into one score — being better at every individual step, and being able to recover after something goes wrong — even though those two carry completely different weight in real deployments.

A question hidden behind the usual metric

The work comes from the Key Laboratory of Cognition and Decision Intelligence for Complex Systems at the Institute of Automation, Chinese Academy of Sciences, the School of Artificial Intelligence at the University of Chinese Academy of Sciences, and Tsinghua University. It is public on arXiv, with a project page and an MIT-licensed GitHub repository released alongside.

The move is to generalize the single-turn Pass@K metric into a two-dimensional Pass@(K,T): K is sampling breadth, meaning how many independently sampled trajectories you get, and T is interaction depth, meaning how many interaction turns each trajectory is allowed. Experiments span four benchmarks — DeepCoder-MT, BFCL, InterCode-SQL, and FrozenLake — using Qwen3-4B and Qwen3-8B as base models and running GRPO, GiGPO, and PPO.

First counterintuitive result: tighten the budget and the advantage flips

Fixing K=8 and sweeping T, the paper’s tables tell a clear story. The RL model’s margin over the base model grows as the interaction budget loosens, but it collapses as the budget tightens, and on some tasks it inverts outright.

The sharpest cell sits at the 25% budget level of BFCL-v3: the Qwen3-4B base model scores 3.0, while full-budget GRPO RL manages only 2.5 — below the base model. FrozenLake behaves the same way at that tier, with base at 30.3 and full-budget GRPO at 28.7. The authors also compute a structural upper bound for each task at each budget, to rule out the trivial explanation that tight budgets simply make tasks unsolvable. Even with substantial headroom remaining below that upper bound, RL still shows no advantage.

Second counterintuitive result: single-step actions are not uniformly better

The authors evaluate at fine granularity on matched states. They sample intermediate states in equal numbers from base and RL trajectories to avoid state-visitation bias, draw G=16 next-step actions from each policy at the same state, and score them by action value — exact Q* values from finite-horizon dynamic programming for FrozenLake with a discount factor of 0.95, and unit-test pass rates for DeepCoder-MT.

The distribution does not favor RL; it leans slightly toward the base model. RL picks the better single-step decision on only 46.8% of DeepCoder-MT states and 42.8% of FrozenLake states, with average quality no better than the base model and in fact slightly below it.

But the strength is unmistakable: error recovery

Change the framing and the picture inverts. The authors build an error-state pool by injecting an incorrect action at the preceding step. Here, RL selects the superior action in 65.7% of FrozenLake states and 67.3% of DeepCoder-MT states. Continuing from identical error states, terminal success rates favor RL at 51.3% against 38.6% on FrozenLake, and 38.5% against 22.9% on DeepCoder-MT.

The BFCL case study is the most engineering-flavored result. They collect 191 trajectory prefixes containing tool-call errors — 91 from RL rollouts, 100 from base — and let both models take one action from each identical prefix. RL’s error correction rate is 41.4% versus 27.2% for the base model. The appendix also notes that RL’s advantage is not evident early in a trajectory and can even trail the base model, only widening as interaction proceeds.

Mechanism: locally plausible steps, a different policy over whole trajectories

The likelihood analysis explains why. Evaluating the negative log-likelihood of single-step actions under the base policy gives 12.20 for RL against 10.25 for base, with the two distributions overlapping heavily — RL’s individual actions do not stray outside the base model’s support. But switch to the cumulative log-likelihood ratio over a complete trajectory, and successful RL trajectories reach roughly 228 while successful base trajectories reach only 26.

The paper then adds a length-matched control, requiring output token counts to differ by at most 10% of the longer sequence: the matched difference is 185.9 nats on Terminal and 805.8 nats on BFCL, with both 95% confidence intervals above zero — 193 pairs across 89 tasks on Terminal, 283 pairs across 146 tasks on BFCL. In other words, the policy shift at the trajectory level is not simply a byproduct of longer outputs.

Training dynamics nail the point down. On DeepCoder-MT, the trajectory-level reward Rmulti rises by 0.331 while the single-step reward Rsingle drops by 0.097. The authors phrase it as a pattern that grows progressively during training rather than an accident at the final checkpoint.

Two mitigations, one of them deployable

The first path is process supervision signals. On FrozenLake, exact Q-value process rewards and GiGPO stabilize performance under tight budgets. On BFCL, where no exact step-level oracle exists, neither GiGPO nor PPO beats GRPO under constrained budgets and simply matches it. The paper’s conclusion here is pessimistic: such supervision is rarely accessible in realistic agent tasks.

The second is the paper’s own BudgetMix RL, a deliberately small change — the reward mechanism stays the same, only the budget distribution moves. For each task, candidate budgets are constructed at 10% intervals and sampled uniformly, with all trajectories in a group sharing one budget; during training the total and remaining budget are written into the prompt, and any trajectory that exhausts its budget without finishing receives zero reward. The paper reports average gains over full-budget RL of 7.5, 5.3, and 4.0 percentage points at the 25%, 50%, and 75% tiers respectively, while maintaining parity at 100%. Averaging the per-task gain rows by model size yourself reproduces the figures: at the 25% tier, 6.6 and 8.35 average to 7.475, matching the reported 7.5; the 50% tier gives 5.3125 against 5.3, and the 75% tier gives 4.0125 against 4.0 — all three recompute cleanly.

Two hard gates before you ship this

The first is how the budget is communicated. Every experiment in the paper puts the total and remaining budget explicitly into the system prompt, but in real deployments the cap is usually a timeout or a cost ceiling that is never announced. A reader on the paper’s page asks this directly: if the cap is not announced, does the model learn a generally more careful early policy, or does it merely condition on the stated number? The paper does not answer.

The second is the reproduction boundary. The official repository states plainly that its examples and engineering checks are not numerical reproductions of the paper’s benchmarks, and that model weights, private benchmark tests, original traces, and full experiment results are not bundled. What the repo does provide is a diagnostic toolchain: tight-budget evaluation, error-prefix evaluation, training-dynamics tracking, BudgetMix planning and auditing, plus likelihood and process-signal method paths.

So what

If your agent has an interaction-turn cap in production, the evaluation metric should grow a Pass@T dimension rather than reporting task success alone — by the paper’s own tables, the same model can flip from ahead to behind simply by moving budget tier. And if the task mixes in irreversible actions like financial operations or database writes, single-step decision quality is the thing that genuinely deserves scrutiny, and multi-turn RL offers no guarantee there. What it buys is resilience: the ability to climb back after a mistake.

So the accurate phrasing is not "multi-turn RL makes agents smarter." It is closer to "multi-turn RL makes agents harder to break, given enough interaction budget."

Sources: paper https://arxiv.org/abs/2611.06483 , project page https://rethinking-agentic-rl.github.io/ , official repository https://github.com/GaryStack/Rethinking-Agent-RL .