[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-agentopsd-recursive-belief-credit-assignment":3,"news-related-777afb24-262f-45cc-961f-d5d49ad42883":41},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"777afb24-262f-45cc-961f-d5d49ad42883","AgentOPSD 用递归贝叶斯信念破解多轮 Agent 强化学习的信用分配：清华\u002F浙大\u002F美团让 GRPO 学会看哪个 turn 决定胜负","清华、浙大与美团联合提出的 AgentOPSD，把每轮 token 级 teacher-student 概率差聚合成 turn 级证据，并在 log-odds 空间递归更新「轨迹成功」的贝叶斯信念；该方法无需 critic 网络或额外 rollout，即可在 ALFWorld、WebShop 与 Search-QA 上用 Qwen2.5-3B\u002F7B 一致性跑赢 GRPO 与 5 个自蒸馏基线，并配套给出四维消融与超参敏感性曲线。","# AgentOPSD：用递归贝叶斯信念，让 GRPO 在多轮 Agent 上学会区分「关键 turn」与「走过场 turn」\n\n## 1. 多轮 Agent RL 的老问题：trajectory 优势被广播，长 horizon 必然失真\n近一年 GRPO 系列工作已经稳定把「RL with verifiable rewards」推到了 LLM 数学、代码、GUI、Web 自动化与多模态搜索等场景。但只要任务变成多轮 agent 交互，稀疏 terminal reward 就会让信用分配变得格外困难。GRPO 在每个 group 内把序列级 advantage **A_seq** 直接广播到所有 token；长轨迹里「关键决策」和「例行公事」被同一信号驱动，模型当然学不到「哪一步决定胜负」。GRPO+OPSD、Skill-SD、RLSD、SDAR、StepOPSD 等近期工作都试图把 teacher-student 的对数概率差嵌进优势，但它们的共同假设是：turn 之间是独立的，**单点的 local gap 就能代表 sequential credit**。\n\n## 2. AgentOPSD 的核心思路：local gap 不是 sequential credit\n来自清华（Zi-Han Wang、Jinyang Wu 等）、浙大（Zhengxi Lu 等）和美团（Qi Gu、Xunliang Cai 等）的 AgentOPSD（[arXiv:2608.05987](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.05987)）把这件事拆到三步：\n\n1. **把 token 级 teacher-student log-prob gap 聚合成 turn 级 evidence**：在每个环境切换边界求和，得到 e_k = log π_θ(a_k | s_k, c⁺) − log π_θ(a_k | s_k)，其中 c⁺ 是训练阶段从 SkillRL SkillBank 里按关键词检索出的私有 skill 描述；\n2. **递归更新成功信念 B_k**：用 group 内成功率 R̄ 当先验 B_0 = clip(R̄, ε₀, 1−ε₀)，按 c_k = γ c_{k-1} + e_k 在 log-odds 空间累加，B_k = σ(logit(B_0) + c_k)；\n3. **用边际变化 ΔB_k = B_k − B_{k-1} 替代原始 gap**：再叠上 outcome 符号 sign(A_seq) 得到 q_k，做 bounded multiplier w_k = clip(1 + b·z_k, 1−b, 1+b) 重塑优势。\n\n文章核心论点就一句话：**「一个孤立的 local gap 不足以成为 sequential credit，真正决定 credit 的是该 gap 在历史累积信念上的边际变化。」** 这条思路不依赖 critic、value network 或额外 rollout，代价只是一次额外的 teacher forward pass。\n\n## 3. 主要数据：三个环境、两个尺度，全部刷掉 GRPO 与 5 个自蒸馏基线\n在 Qwen2.5-7B-Instruct 上，AgentOPSD 把 ALFWorld 平均成功率从 GRPO 的 81.2% 拉到 **89.1%**，WebShop Score 从 80.9 拉到 **90.2**、Accuracy 从 72.6 拉到 **79.7**；Search-QA 在 Qwen2.5-3B 上的平均分从 GRPO 的 36.4 升到 **46.7**。更关键的是 horizon 越长 AgentOPSD 越能扛：ALFWorld 上把「每多一个 turn 损失的成功率」压到 −0.54pp，而 RLSD 是 −3.59pp、GRPO 是 −2.91pp，差距是数量级的。\n\n机制消融表（Table 2）验证了四个组件缺一不可：把 recursive 改成 raw local gap e_k 直接掉 6.3 点；把 outcome sign 去掉掉 8.6 点；把经验先验 B_0 去掉掉 10.2 点；把 turn-level 聚合换成 per-token 掉 3.2 点。这组消融把「为什么是 recursive belief 而不是 score averaging」讲得很死。\n\n## 4. 工程侧值得注意的三件事\n- **超参只有 λ、γ、ε_high 三个**，λ=0.5、γ=0.95、ε_high=0.24 在 ALFWorld \u002F Search-QA \u002F WebShop 三个环境都跑得稳；γ 在 0.8~1.0 之间结果只差几个点，说明「近期 turn 比远期更重要」是软信号不是硬约束。\n- **代码与训练脚本即将开源**（`github.com\u002FZethWang\u002FAgentOPSD`，Apache-2.0 协议），基于 8×H800 训练 Qwen2.5-3B\u002F7B-Instruct，门槛对工业界很友好。\n- **不与 GiGPO 互斥**：GiGPO 用重复 anchor 估计 step 级 advantage，AgentOPSD 用 self-distillation 估计 evidence；前者是 reward-side credit，后者是 signal-side credit，理论上可以叠加。\n\n## 5. 所以呢\n如果你在做多轮 agent 训练（尤其是 Web 自动化、GUI、SearchQA、ALFWorld 这样的长 horizon 任务），AgentOPSD 给的是「**不用引入 critic，就能把 group-relative RL 的均匀优势信号拆成 turn-level dense supervision**」的工程答案。它背后那套「把 credit 当成 belief revision 来做」的视角，也给 OPSD 系列后续工作（RUDDER、VinePPO、process reward model）提供了一种更轻的解释。\n\n但也别被「Qwen2.5-3B 也刷到 84.4%」这种数据带偏：它仍然依赖 SkillRL 的私有 skill 检索作为 teacher 分支，自部署团队需要先把技能库搭起来，这一层的工程量没在论文里展开。\n\n**资料来源**：AgentOPSD 原论文 [arXiv:2608.05987](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.05987)（Zi-Han Wang 等，清华+浙大+美团，2026-08-06 v1）；代码仓库 `github.com\u002FZethWang\u002FAgentOPSD`；对比基线 RLSD、SDAR、StepOPSD、GRPO+OPSD、Skill-SD、OPSD、Skill-GRPO 均出自同一论文 Table 1。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.05987","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":25,"name":26,"slug":26,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"1b50e509-c72e-4fa2-885e-0d699224dc42","en","AgentOPSD: recursive Bayesian credit for multi-turn agent RL","A joint Tsinghua–Zhejiang University–Meituan team proposes AgentOPSD, which aggregates token-level teacher–student log-probability gaps into turn-level evidence and recursively updates a Bayesian success-belief state in log-odds space; without a critic or extra rollouts, it consistently beats GRPO and five strong self-distillation baselines on ALFWorld, WebShop and Search-QA with Qwen2.5-3B\u002F7B, and ships a four-axis ablation plus hyper-parameter sensitivity analysis.","# AgentOPSD: Recursive Bayesian Beliefs Fix Turn-Level Credit Assignment for Multi-Turn Agentic RL\n\n## 1. The old problem: trajectory-level advantage gets broadcast, so long-horizon agent RL is structurally blind\nGRPO-style reinforcement learning with verifiable rewards has reached math, code, GUI, web automation and multimodal search. The moment the task becomes a multi-turn agent loop, however, sparse terminal rewards break credit assignment. GRPO broadcasts the group-relative sequence-level advantage **A_seq** uniformly to every token in a trajectory, so the long-tail of \"routine\" turns drowns out the few \"pivotal\" decisions. A recent wave of on-policy self-distillation methods (GRPO+OPSD, Skill-SD, RLSD, SDAR, StepOPSD) tries to inject the teacher–student log-prob gap into the advantage, but they all share one assumption: turns are independent, so a local gap already represents sequential credit.\n\n## 2. The AgentOPSD idea: a local gap is not sequential credit\nAgentOPSD (Tsinghua's Zi-Han Wang \u002F Jinyang Wu, Zhejiang University's Zhengxi Lu, Meituan's Qi Gu \u002F Xunliang Cai, [arXiv:2608.05987](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.05987)) breaks the problem into three steps:\n\n1. **Aggregate token-level teacher–student log-prob gaps at turn boundaries** to obtain evidence `e_k = log π_θ(a_k | s_k, c⁺) − log π_θ(a_k | s_k)`, where the privileged branch sees a skill description `c⁺` retrieved from the SkillRL SkillBank at training time only.\n2. **Recursively update a trajectory-success belief `B_k`** in log-odds space: start from the group success rate `B_0 = clip(R̄, ε₀, 1−ε₀)`, accumulate `c_k = γ c_{k-1} + e_k` and squash with `B_k = σ(logit(B_0) + c_k)`.\n3. **Use the marginal revision `ΔB_k = B_k − B_{k-1}` as the per-turn credit**; multiply by `sign(A_seq)` to keep the verifier-aligned direction, and apply a bounded multiplier `w_k = clip(1 + b·z_k, 1−b, 1+b)` to reshape the advantage.\n\nThe paper's central slogan: **a local gap, in isolation, is not sequential credit; what matters is how much that gap revises the belief state accumulated over the preceding history.**\n\n## 3. Headline numbers: ALFWorld 89.1%, WebShop 90.2, Search-QA +10 points\nOn Qwen2.5-7B-Instruct, AgentOPSD lifts ALFWorld average success from GRPO's 81.2% to **89.1%**, WebShop Score from 80.9 to **90.2** and Accuracy from 72.6 to **79.7**; Search-QA on Qwen2.5-3B jumps from GRPO's 36.4 to **46.7**. The gain also scales with horizon: the success-rate loss per additional turn on ALFWorld drops to −0.54pp for AgentOPSD, vs. −3.59pp for RLSD and −2.91pp for GRPO.\n\nThe four-axis ablation in Table 2 confirms that every piece is load-bearing: swapping the recursive revision for the raw local gap `e_k` costs 6.3 points; dropping the outcome sign costs 8.6; replacing the empirical prior `B_0` costs 10.2; falling back to per-token accumulation costs 3.2. Together they lock in the \"credit is a belief revision, not a score average\" claim.\n\n## 4. Three engineering takeaways\n- **Only three real hyperparameters** (`λ`, `γ`, `ε_high`); `λ=0.5`, `γ=0.95`, `ε_high=0.24` is robust across ALFWorld \u002F Search-QA \u002F WebShop. Sweeping `γ ∈ [0.8, 1.0]` only changes results by a few points, so \"recent turns matter more than far-back turns\" is a soft prior, not a hard constraint.\n- **Code and training scripts are promised on `github.com\u002FZethWang\u002FAgentOPSD` (Apache-2.0)**. The paper trains Qwen2.5-3B\u002F7B-Instruct on 8×H800 GPUs, so the recipe is well within reach for a typical industrial lab.\n- **Complementary to GiGPO**: GiGPO estimates step-level advantages via repeated anchor states; AgentOPSD estimates them via self-distillation evidence. They use orthogonal signals (reward-side vs. signal-side) and can in principle stack.\n\n## 5. So what\nFor anyone training multi-turn agents (web automation, GUI control, retrieval-augmented QA, embodied household tasks), AgentOPSD offers a \"critic-free way to convert a group-relative uniform advantage into dense turn-level supervision.\" The \"credit is a belief revision\" framing also gives OPSD-family work (RUDDER, VinePPO, process reward models) a lighter-weight way to think about the same problem.\n\nThe \"Qwen2.5-3B reaches 84.4%\" headline should be read with one caveat: the teacher branch still relies on a SkillRL-style private skill retrieval, so production teams need to build a skill corpus first. The engineering cost of that pre-step is not in the paper.\n\n**Sources**: AgentOPSD paper, [arXiv:2608.05987](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.05987) (Zi-Han Wang et al., Tsinghua + Zhejiang University + Meituan, 2026-08-06 v1); code at `github.com\u002FZethWang\u002FAgentOPSD`; baselines (RLSD, SDAR, StepOPSD, GRPO+OPSD, Skill-SD, OPSD, Skill-GRPO) all from the same paper's Table 1.\n","agentopsd-recursive-belief-credit-assignment","2026-08-07T02:00:00Z","2026-08-09T16:27:47.632877Z","2026-08-09T16:27:47.632887Z",true,"agent",155,{"items":42},[43,48,53,58,63,68],{"id":44,"title":45,"news_slug":46,"published_at":47},"22a1a718-0eb6-46e5-8ee8-825400de11d1","DeepMind WeatherNext 在 Nature 发论文：用 28 km 粗分辨率做出多一天的飓风预警,代码权重全部开源","deepmind-weathernext-cyclones-nature-open-source","2026-08-10T02:00:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"af056e63-5622-48ae-8629-5226aed64767","PalmClaw 把端侧 Agent 拉进「原生」时代:94.9% 完成时间压缩 + 11.5% 成功率提升","palmclaw-on-device-agent","2026-07-15T20:10:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"5083a7bf-ab57-4ddc-900e-096af6d618d0","AutoTool 把工具调用做成「动态选择」:训练见 460 工具,推理泛化到 1346 个工具","autotool-dynamic-tool-selection","2026-07-12T14:10:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"12df58ff-0771-4c0e-bf0e-00bfdc8112bb","SkillCenter：21 万可审计 Agent 技能库，SQLite 离线检索","skillcenter-sqlite-agent","2026-07-09T04:30:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"1316635c-88e1-41b6-a45c-df8ef217cf3f","PaperPilot 把文献搜索改写成「工作流归纳」：可编辑 DAG 把多轮检索错误率干到 0%","paperpilot-workflow-induction","2026-07-01T08:21:23+00:00",{"id":69,"title":70,"news_slug":71,"published_at":72},"6de305c2-91b2-47fb-b1e4-dfb5f1e711c8","WorldEvolver：把世界模型装进 LLM Agent 的「即时记忆」","worldevolver-llm-agent-world-model","2026-06-30T18:04:00+00:00"]