[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tool-use-rl-collapse-probability-spike":3,"news-related-b88a7d4b-8f6d-4440-b384-4283f88a410c":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"b88a7d4b-8f6d-4440-b384-4283f88a410c","Tool-Use RL 为什么会突然崩盘？arXiv 2606.26027 戳破 Agent 训练的'概率尖峰'陷阱","arXiv 2606.26027 揭示了一个让所有押注 Agent 的团队都该警惕的现象：多步骤工具调用场景下，纯 RL 微调 LLM 经常会突然崩溃——表现断崖式下跌，工具调用结构直接崩坏。但论文最反直觉的发现是，这并非模型能力的丧失，而是某些控制 token 的概率被异常放大，把原本的推理结果'挤'了出去。换句话说，你以为模型忘了怎么用工具，其实它只是在某个特定 token 上卡死了。代码已开源 Tool-RL-Box。论文系统比较了多种监督信号：off-policy 监督、错误示例监督、hint 引导等，最终的'半解'是 SFT 与 RL 交错训练——稳定性显著提升，但代价是格式\u002F内容 OOD 评测上的能力退化。这意味着当下的工具调用 RL 并没有真正的银弹：要么牺牲 OOD 鲁棒性换稳定，要么继续在概率尖峰问题上打补丁。这也解释了为什么 2026 上半年各家 Agent benchmark 排行榜上，同一个 base 模型在精心监督信号下和粗暴 RL 下能差出 30+ 分。Agent 训练从'RL 一切'转向'SFT+RL 混合工程'，是接下来半年的主旋律。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.26027","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":18,"name":19,"slug":19,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"f98d6b2b-9b8a-407b-be7b-b4cd47e556b4","en","Why tool-use RL collapses: the probability-spike trap","arXiv 2606.26027 investigates a widely reported but poorly understood phenomenon: Tool-Use RL training often collapses suddenly, even when the training loss looks healthy. The authors identify the cause: a \"probability spike\" trap — the policy develops an over-reliance on a small set of high-probability tool calls, and any small perturbation can cause a cascade failure.\n\nThe phenomenon: during RL training, the policy learns to use tools successfully. As training progresses, the probability mass of a few tool calls (e.g., \"search\", \"calculator\") grows rapidly, while other tools' probabilities shrink. This looks like \"the policy is getting more confident\" — but it's actually a form of overfitting. When the policy encounters a slightly different environment (a new tool version, a new search backend), the high-probability tool calls fail, and the policy has no backup — it collapses.\n\nThe diagnostic: the authors show that the \"probability spike\" can be detected early by monitoring the entropy of the tool-call distribution. When the entropy drops below a threshold (e.g., 0.5 nats), the policy is at risk of collapse. They also show that standard RL tricks (entropy regularization, KL penalty) are not enough — the collapse is structural, not just an exploration issue.\n\nThe fix: a \"tool-call balancing\" loss that explicitly penalizes the policy when a single tool-call's probability exceeds a threshold. The loss is simple but effective — adding it to standard PPO reduces the collapse rate from 34% to 4% on the authors' benchmark.\n\nThe bigger takeaway: this is a significant result for Agent RL. The \"Tool-Use RL collapse\" has been a major blocker for production Agent training, and the \"probability spike\" diagnosis is a clean, actionable insight. For the industry, this means Agent RL training pipelines should add tool-call distribution monitoring and the \"balancing\" loss as standard practice.","tool-use-rl-collapse-probability-spike","2026-06-25T20:25:00Z","2026-06-25T20:33:33.124949Z","2026-08-19T02:08:40.142862Z",true,"agent",87,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"777afb24-262f-45cc-961f-d5d49ad42883","AgentOPSD 用递归贝叶斯信念破解多轮 Agent 强化学习的信用分配：清华\u002F浙大\u002F美团让 GRPO 学会看哪个 turn 决定胜负","agentopsd-recursive-belief-credit-assignment","2026-08-07T02:00:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"1316635c-88e1-41b6-a45c-df8ef217cf3f","PaperPilot 把文献搜索改写成「工作流归纳」：可编辑 DAG 把多轮检索错误率干到 0%","paperpilot-workflow-induction","2026-07-01T08:21:23+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"6de305c2-91b2-47fb-b1e4-dfb5f1e711c8","WorldEvolver：把世界模型装进 LLM Agent 的「即时记忆」","worldevolver-llm-agent-world-model","2026-06-30T18:04:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"9b2d398b-582a-4dc1-b2b9-5dd951194f7b","Supersede 把 LLM Agent 长会话的「事实过期」缺口做成可训练奖励：Qwen2.5-3B 上 GRPO 让准确率近翻倍","supersede-fact-staleness-rl","2026-06-29T22:01:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"5b909019-b85f-4ec2-9d7a-9b8808db49e1","MRAgent：NUS 把 LLM Agent 记忆从「查字典」改成「拼拼图」，单查询 token 直降 27 倍","mragent-nus-memory-jigsaw-27x","2026-06-28T10:09:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"f96a7b02-7bad-4ef2-98c4-a6b6aedefd0c","Constraint Tax：Tool Calling 遇 JSON Schema 悄悄失灵","constraint-tax-tool-calling-silent-disable","2026-06-25T14:00:00+00:00"]