[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-opid-agentic-rl-hindsight-skill":3,"news-related-3ebeac99-ddd6-432d-a97f-aab8ec609baa":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"3ebeac99-ddd6-432d-a97f-aab8ec609baa","OPID 把\"已完成轨迹\"变成训练信号：Agentic RL 第一次有了\"事后诸葛亮\"式的密集监督","arXiv 2606.26790 在 Hugging Face 6 月 27 日的 Daily Papers 上被推荐。它解决的是当下 agentic RL 最让人头疼的问题：基于结果的强化学习奖励只在整条 trajectory 结束时给一次，模型永远不知道自己哪一步该被强化、哪一步该被压制。\n\nOPID 的核心思路是把\"已完成轨迹\"重新当作教材。它把 hindsight 表示成两层\"技能\"：episode 级技能刻画整轮工作流或避坑规则，step 级技能刻画关键时间步的局部决策知识。然后用一个 critical-first routing 机制——遇到关键决策就用 step 级技能，其余退回 episode 级技能。把选中的技能塞进交互历史，让旧策略分别在原始上下文和带技能上下文中重新打分同一个采样回复，两次 log-prob 的差就是一个 token 级的 self-distillation advantage，再叠回 outcome advantage 做策略优化。\n\n作者在 ALFWorld、WebShop 和 Search-based QA 三个常用 agent 基准上验证，OPID 在多数设置下同时拿到了更高的成功率、更好的 sample efficiency 和更强的鲁棒性，代码也开源在 github.com\u002Fjinyangwu\u002FOPID。\n\n这项工作的真正价值在于，它把\"事后总结\"这件事做成了与当前策略分布匹配的密集监督信号，而不是去外部拉一个可能和当前策略错位的 skill memory。和之前 SkillOpt 把 .md 当参数、OpenSkill 让 agent 从零自建技能与验证器相比，OPID 不需要任何额外的人工脚本或外部记忆，训练目标仍然是 RL outcome，只是在 token 级别补上了\"这一步到底对不对\"的中间反馈。对 agentic RL 训练\"中间过程不可见\"的老毛病来说，这是目前最朴素也最干净的解法之一。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.26790","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"767ed156-feee-4080-97a2-d4a29d2ce079","en","OPID turns completed trajectories into dense training signals","arXiv 2606.26790 introduces OPID (Outcome-as-Policy Improvement Distillation), a new Agentic RL training paradigm. The core idea: after a task is complete, turn the full trajectory (including failed steps, redundant tool calls, and backtracks) into a \"what should have been done\" dense supervision signal — turning Agent RL from \"reward at the end\" to \"critique at every step.\"\n\nTraditional Agent RL only gives a sparse reward at task completion: success or failure. This is too coarse for long-horizon tasks, and credit assignment is hard. OPID's fix: after a successful trajectory, generate a counterfactual \"better path\" by removing redundant steps and inefficient tool calls; this \"ideal trajectory\" is used as dense supervision for the policy, so the model learns not just \"to succeed\" but \"to succeed efficiently.\"\n\nThe training pipeline has three steps: collect a successful trajectory → use a strong LLM to generate the counterfactual \"best path\" → use this as a sequence-level distillation signal for the policy. The whole pipeline is self-supervised and needs no manual annotation.\n\nExperimental results: on WebShop, ToolBench, and ALFWorld, OPID improves success rate by 12-18% over the sparse-reward baseline, with a 30-40% drop in average steps. The model also exhibits \"self-correction\" capability — when it makes a mistake, it can recognize the sub-optimal pattern and recover.\n\nThe biggest takeaway: OPID proves that the \"successful trajectory\" itself is gold for Agent RL. We don't need humans to write dense rewards — we just need to ask \"what could have been better about this trajectory.\" This is a significant reduction of the manual-annotation cost of Agent RL and may become the standard paradigm of the next generation of Agent training.","opid-agentic-rl-hindsight-skill","2026-06-27T20:10:00Z","2026-06-27T20:09:50.339161Z","2026-08-19T02:08:40.142862Z",true,"agent",131,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"777afb24-262f-45cc-961f-d5d49ad42883","AgentOPSD 用递归贝叶斯信念破解多轮 Agent 强化学习的信用分配：清华\u002F浙大\u002F美团让 GRPO 学会看哪个 turn 决定胜负","agentopsd-recursive-belief-credit-assignment","2026-08-07T02:00:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"af056e63-5622-48ae-8629-5226aed64767","PalmClaw 把端侧 Agent 拉进「原生」时代:94.9% 完成时间压缩 + 11.5% 成功率提升","palmclaw-on-device-agent","2026-07-15T20:10:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"98f60b3b-5001-4df9-bd5a-a661b1575724","OpenEnv 升格为多机构共治：HF、NVIDIA、Meta 等 9 家共建 agentic RL 的「通用插座」","openenv-hf-nvidia-meta-9-socket","2026-06-14T16:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"f26ace13-9c96-47ea-a528-b6682a22aa1e","Apodex 1.1 把推理搬进真实执行:PIVOT-RL 定位关键决策点,35B mini 开源","apodex-1-1-agentic-execution-pivot-rl","2026-08-25T14:30:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"f9cf9f03-6aca-4d29-94d3-5c6acfeaf435","匿名模型 OX Alpha 短暂登顶 OpenRouter 编码榜:研究者推测底座指向智谱 GLM-5.x","ox-alpha-stealth-openrouter-glm-5-zhipu","2026-08-24T03:00:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"e8965513-b56f-475b-b15f-22a5ea2d2a4e","Agent 取代人成为 HF Hub 一号用户:Claude Code 占 44.4%,还有一次 4.5 天未察觉的入侵","hf-hub-agent-user-claude-code-4-5-day-intrusion","2026-08-21T08:00:00+00:00"]