Reinforcement-learning recipes for long-horizon LLM agents mostly inherit a default from single-turn RL: treat every generated token as an action. PPO-style methods estimate advantages along the token stream with a learned critic; GRPO-style group-relative methods hand one shared advantage to an entire sequence. A paper posted October 26 — Multiscale Policy Optimization (MPO) — starts by measuring whether that default matches how value actually behaves in multi-turn agents, and the measurement challenges it (arXiv:2610.35013).
Measure first
The authors instrumented a search-agent training run: 2,048 trajectories and 16,323 turns across eight checkpoint batches. Critic values stay relatively stable inside a turn and jump at turn boundaries. Quantified, differences between turn means account for a median 89.5% of the critic's token-weighted within-trajectory value variance, and between-turn variance exceeds within-turn variance in 93.8% of trajectories. The implication the paper draws: the environment acts on complete assistant turns — a search query, a tool call, a final answer — so the macro decision unit is the turn, not the token. Failed and successful trajectories in Figure 2 show the same shape.
The gap between two families
The paper groups prior methods into two families. Critic-free group-relative methods (GRPO, DAPO, GSPO) derive one shared advantage per rollout and cannot distinguish which turn or token deserved the credit. Token-level critic methods (PPO, VC-PPO, VAPO, SAO) estimate finer advantages but never separate within-turn generation from between-turn interaction. MPO's position is that both scales are needed: propagate credit across turns, then refine it within each turn.
Three components
First, Multiscale Advantage Estimation (MAE): treat each complete turn as a macro action and run GAE on the turn axis, so credit decays once per interaction rather than once per token, independent of turn length; a within-turn local trace, centered, then refines token credit while locking the turn mean, with β scaling the local correction. Second, turn-level importance sampling: forward weighting and clipping use the complete-turn probability ratio, while a stop-gradient construction lets every token keep its own policy gradient. Third, the critic regresses the terminal return directly (λcritic=1, following VAPO's decoupled GAE). The ablated best setting is (λ, λintra, β) = (0.80, 0.95, 1.00).
The scoreboard
Experiments use Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct across search (BrowseComp-Plus, GAIA, HLE text-only subsets), tool use (Claw-Eval) and automation (Automation Bench), against GRPO, GSPO, PPO, VAPO and SAO under matched budgets. On 4B, BrowseComp-Plus pass@1 reaches 48.08 versus 43.75 for the strongest baseline SAO (+4.33); GAIA 39.47 (+3.85) and HLE 9.92 (+0.66). On 30B-A3B the three pass@1 scores are 50.32 / 43.42 / 10.03, leading by 3.59 / 3.80 / 0.12. Claw-Eval pass3 gains 0.95 and 1.12 points; Automation Bench climbs from a 0.00 base to 13.33 on 4B and 15.00 on 30B. In the paper's own evaluation MPO tops all nine metrics at both scales, and the edge holds as sampling grows: 4B pass@32 hits 75.90 on BrowseComp-Plus. Two ablations carry the argument. Swapping turn-level ratios back to token-level costs 2.36, 3.12 and 1.03 pass@1 on the 4B search benchmarks; removing within-turn allocation (β=0) drops BrowseComp-Plus pass@1 by 6.28 — yet still beats PPO.
The cold water
All numbers are self-reported, with no third-party reproduction yet; the evaluation protocol selects the best checkpoint per run and averages five offline evaluations, and each ablation setting has a single run. As of writing, the paper has released no code or models — the Hugging Face paper page links none. Community comments raise a practical concern: with wildly unequal turn lengths, a 40-token function call against a 6,000-token file read, turn-level ratios carry very different variance, and clipping may need per-task retuning; the paper does not answer that directly. The paper sits under the IQuest badge on Hugging Face's Daily Papers at #2 of the day with 315 upvotes; the same group shipped the open-weight CLI agent model IQuest-Q1 a week earlier — model first, training method next, a rhythm worth watching. The lane is also getting crowded: AgentOPSD (August) taught GRPO to spot the decisive turn via recursive Bayesian beliefs — a different mechanism converging on the same turn-level insight.
So what
The most transferable asset here may be the 89.5% number, not the algorithm. If your own critic is also near-constant within turns, most of your token-level advantage precision is probably noise — worth auditing before the next training run. One commenter suggests reusing the turn-level critic value at inference as a cheap anomaly score: a sudden deviation within a turn triggers a retry or a human check. Until the code lands, this belongs on the watchlist, not in your training pipeline.