AgentOPSD: Recursive Bayesian Beliefs Fix Turn-Level Credit Assignment for Multi-Turn Agentic RL

1. The old problem: trajectory-level advantage gets broadcast, so long-horizon agent RL is structurally blind

GRPO-style reinforcement learning with verifiable rewards has reached math, code, GUI, web automation and multimodal search. The moment the task becomes a multi-turn agent loop, however, sparse terminal rewards break credit assignment. GRPO broadcasts the group-relative sequence-level advantage A_seq uniformly to every token in a trajectory, so the long-tail of "routine" turns drowns out the few "pivotal" decisions. A recent wave of on-policy self-distillation methods (GRPO+OPSD, Skill-SD, RLSD, SDAR, StepOPSD) tries to inject the teacher–student log-prob gap into the advantage, but they all share one assumption: turns are independent, so a local gap already represents sequential credit.

2. The AgentOPSD idea: a local gap is not sequential credit

AgentOPSD (Tsinghua's Zi-Han Wang / Jinyang Wu, Zhejiang University's Zhengxi Lu, Meituan's Qi Gu / Xunliang Cai, arXiv:2608.05987) breaks the problem into three steps:

  1. Aggregate token-level teacher–student log-prob gaps at turn boundaries to obtain evidence e_k = log π_θ(a_k | s_k, c⁺) − log π_θ(a_k | s_k), where the privileged branch sees a skill description c⁺ retrieved from the SkillRL SkillBank at training time only.
  2. Recursively update a trajectory-success belief B_k in log-odds space: start from the group success rate B_0 = clip(R̄, ε₀, 1−ε₀), accumulate c_k = γ c_{k-1} + e_k and squash with B_k = σ(logit(B_0) + c_k).
  3. Use the marginal revision ΔB_k = B_k − B_{k-1} as the per-turn credit; multiply by sign(A_seq) to keep the verifier-aligned direction, and apply a bounded multiplier w_k = clip(1 + b·z_k, 1−b, 1+b) to reshape the advantage.

The paper's central slogan: a local gap, in isolation, is not sequential credit; what matters is how much that gap revises the belief state accumulated over the preceding history.

3. Headline numbers: ALFWorld 89.1%, WebShop 90.2, Search-QA +10 points

On Qwen2.5-7B-Instruct, AgentOPSD lifts ALFWorld average success from GRPO's 81.2% to 89.1%, WebShop Score from 80.9 to 90.2 and Accuracy from 72.6 to 79.7; Search-QA on Qwen2.5-3B jumps from GRPO's 36.4 to 46.7. The gain also scales with horizon: the success-rate loss per additional turn on ALFWorld drops to −0.54pp for AgentOPSD, vs. −3.59pp for RLSD and −2.91pp for GRPO.

The four-axis ablation in Table 2 confirms that every piece is load-bearing: swapping the recursive revision for the raw local gap e_k costs 6.3 points; dropping the outcome sign costs 8.6; replacing the empirical prior B_0 costs 10.2; falling back to per-token accumulation costs 3.2. Together they lock in the "credit is a belief revision, not a score average" claim.

4. Three engineering takeaways

  • Only three real hyperparameters (λ, γ, ε_high); λ=0.5, γ=0.95, ε_high=0.24 is robust across ALFWorld / Search-QA / WebShop. Sweeping γ ∈ [0.8, 1.0] only changes results by a few points, so "recent turns matter more than far-back turns" is a soft prior, not a hard constraint.
  • Code and training scripts are promised on github.com/ZethWang/AgentOPSD (Apache-2.0). The paper trains Qwen2.5-3B/7B-Instruct on 8×H800 GPUs, so the recipe is well within reach for a typical industrial lab.
  • Complementary to GiGPO: GiGPO estimates step-level advantages via repeated anchor states; AgentOPSD estimates them via self-distillation evidence. They use orthogonal signals (reward-side vs. signal-side) and can in principle stack.

5. So what

For anyone training multi-turn agents (web automation, GUI control, retrieval-augmented QA, embodied household tasks), AgentOPSD offers a "critic-free way to convert a group-relative uniform advantage into dense turn-level supervision." The "credit is a belief revision" framing also gives OPSD-family work (RUDDER, VinePPO, process reward models) a lighter-weight way to think about the same problem.

The "Qwen2.5-3B reaches 84.4%" headline should be read with one caveat: the teacher branch still relies on a SkillRL-style private skill retrieval, so production teams need to build a skill corpus first. The engineering cost of that pre-step is not in the paper.

Sources: AgentOPSD paper, arXiv:2608.05987 (Zi-Han Wang et al., Tsinghua + Zhejiang University + Meituan, 2026-08-06 v1); code at github.com/ZethWang/AgentOPSD; baselines (RLSD, SDAR, StepOPSD, GRPO+OPSD, Skill-SD, OPSD, Skill-GRPO) all from the same paper's Table 1.