arXiv 2606.26027 investigates a widely reported but poorly understood phenomenon: Tool-Use RL training often collapses suddenly, even when the training loss looks healthy. The authors identify the cause: a "probability spike" trap — the policy develops an over-reliance on a small set of high-probability tool calls, and any small perturbation can cause a cascade failure.
The phenomenon: during RL training, the policy learns to use tools successfully. As training progresses, the probability mass of a few tool calls (e.g., "search", "calculator") grows rapidly, while other tools' probabilities shrink. This looks like "the policy is getting more confident" — but it's actually a form of overfitting. When the policy encounters a slightly different environment (a new tool version, a new search backend), the high-probability tool calls fail, and the policy has no backup — it collapses.
The diagnostic: the authors show that the "probability spike" can be detected early by monitoring the entropy of the tool-call distribution. When the entropy drops below a threshold (e.g., 0.5 nats), the policy is at risk of collapse. They also show that standard RL tricks (entropy regularization, KL penalty) are not enough — the collapse is structural, not just an exploration issue.
The fix: a "tool-call balancing" loss that explicitly penalizes the policy when a single tool-call's probability exceeds a threshold. The loss is simple but effective — adding it to standard PPO reduces the collapse rate from 34% to 4% on the authors' benchmark.
The bigger takeaway: this is a significant result for Agent RL. The "Tool-Use RL collapse" has been a major blocker for production Agent training, and the "probability spike" diagnosis is a clean, actionable insight. For the industry, this means Agent RL training pipelines should add tool-call distribution monitoring and the "balancing" loss as standard practice.