arXiv 2606.26790 introduces OPID (Outcome-as-Policy Improvement Distillation), a new Agentic RL training paradigm. The core idea: after a task is complete, turn the full trajectory (including failed steps, redundant tool calls, and backtracks) into a "what should have been done" dense supervision signal — turning Agent RL from "reward at the end" to "critique at every step."
Traditional Agent RL only gives a sparse reward at task completion: success or failure. This is too coarse for long-horizon tasks, and credit assignment is hard. OPID's fix: after a successful trajectory, generate a counterfactual "better path" by removing redundant steps and inefficient tool calls; this "ideal trajectory" is used as dense supervision for the policy, so the model learns not just "to succeed" but "to succeed efficiently."
The training pipeline has three steps: collect a successful trajectory → use a strong LLM to generate the counterfactual "best path" → use this as a sequence-level distillation signal for the policy. The whole pipeline is self-supervised and needs no manual annotation.
Experimental results: on WebShop, ToolBench, and ALFWorld, OPID improves success rate by 12-18% over the sparse-reward baseline, with a 30-40% drop in average steps. The model also exhibits "self-correction" capability — when it makes a mistake, it can recognize the sub-optimal pattern and recover.
The biggest takeaway: OPID proves that the "successful trajectory" itself is gold for Agent RL. We don't need humans to write dense rewards — we just need to ask "what could have been better about this trajectory." This is a significant reduction of the manual-annotation cost of Agent RL and may become the standard paradigm of the next generation of Agent training.