[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-multi-turn-agentic-rl-budget-dependence":3,"topics-all":38,"news-related-37e0486a-225d-4b44-b189-206bd756210c":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"37e0486a-225d-4b44-b189-206bd756210c","多轮 Agent 强化学习提升了什么:46.8% 的真相","中科院自动化所与清华的论文给多轮 Agent 强化学习做了一次体检:用 Pass@(K,T) 同时衡量采样广度与交互深度后发现,RL 的轨迹级优势依赖充足的回合预算,预算收紧时甚至低于基座;单步决策质量只在 46.8% 的状态上更好,真正的强项是出错之后的恢复能力,而不是把每一步都做得更强。","多轮智能体强化学习几乎成了 Agent 训练的标准配置,但评估口径高度一致:只看任务有没有完成。论文作者指出,只看轨迹结果的评测,把「每一步都做得更好」和「出错之后还能救回来」混成了同一件事——而这两件事在真实部署里的权重完全不同。\n\n## 一个被指标掩盖的问题\n\n论文来自中科院自动化所复杂系统认知与决策智能重点实验室、国科大人工智能学院与清华大学,已在 arXiv 公开,并放出项目页与 MIT 许可的仓库。\n\n他们把单轮 RL 里用惯的 Pass@K 推广成二维的 Pass@(K,T):K 是采样广度,T 是交互深度。实验覆盖 DeepCoder-MT、BFCL、InterCode-SQL、FrozenLake 四个基准,基座是 Qwen3-4B 和 Qwen3-8B,跑 GRPO、GiGPO、PPO。\n\n## 第一个反直觉:预算一收紧,优势就翻转\n\n固定 K=8 扫描 T,论文表格给出的画面很清楚:RL 相对基座的优势随交互预算放宽而越来越明显;预算一收紧,优势迅速缩水,部分任务上甚至反向。\n\n最刺眼的一格在 BFCL-v3 的 25% 预算档:Qwen3-4B 基座是 3.0,GRPO 全预算 RL 反而只有 2.5,比基座还低;FrozenLake 同档,基座 30.3、GRPO 全预算 28.7。论文还算了每个任务在给定预算下的结构性上界,排除「预算太紧、任务本来就无解」的平凡解释:即使上界之下还留着大量空间,RL 依然没有优势。\n\n## 第二个反直觉:单步动作其实没普遍变好\n\n作者在匹配状态上做细粒度评估:从基座和 RL 轨迹等量抽取中间状态避免访问偏差,同一状态下各采 G=16 个下一步动作,用动作价值打分(FrozenLake 用有限时域动态规划算精确 Q* 值,折扣因子 0.95;DeepCoder-MT 用单元测试通过率)。\n\n结论是分布不偏向 RL,甚至略微偏向基座:RL 只在 46.8% 的 DeepCoder-MT 状态和 42.8% 的 FrozenLake 状态上做出更好的单步决策,平均质量不比基座好、反而略低。\n\n## 但强项很明确:错误恢复\n\n换个视角结论就反过来。作者构造错误状态池,在前一步注入错误动作,此时 RL 选出更优动作的比例是 FrozenLake 65.7%、DeepCoder-MT 67.3%。从完全相同的错误状态继续往下跑,终局成功率 RL 是 FrozenLake 51.3% 对基座 38.6%、DeepCoder-MT 38.5% 对 22.9%。\n\nBFCL 上的案例研究更贴近工程直觉:收集 191 条含工具调用错误的前缀(RL 侧 91 条、基座侧 100 条),两个模型从同一条前缀各接一步,RL 的错误纠正率 41.4%,基座只有 27.2%。论文附录还提到,RL 的优势在轨迹早期并不明显,甚至略逊于基座,要随交互推进才逐渐拉开。\n\n## 机制:局部依然合理,轨迹整体换了策略\n\n似然分析给出了解释。在基座策略下评估单步动作的负对数似然,RL 是 12.20、基座是 10.25,两个分布大幅重叠——RL 的单步动作没有跑出基座的支撑集。但换成整条轨迹的累积对数似然比,成功的 RL 轨迹达到约 228,成功的基座轨迹只有 26。\n\n论文还做了长度匹配对照,把两边输出 token 数差控制在长序列的 10% 以内:Terminal 上匹配差 185.9 nats、BFCL 上 805.8 nats,95% 置信区间都高于零(89 个任务 193 对、146 个任务 283 对)。这说明轨迹层面的策略迁移并非单纯由输出更长带来。\n\n训练动态则把这件事钉死了:DeepCoder-MT 上,轨迹级奖励 Rmulti 上涨 0.331 的同时,单步奖励 Rsingle 反而下跌 0.097。论文称这不是终点检查点的偶然,而是训练中逐步长出来的。\n\n## 两个缓解方案,一个能落地一个不能\n\n第一条路是过程监督信号。在 FrozenLake 上,精确的 Q 值过程奖励和 GiGPO 能在紧预算下稳住表现;换到 BFCL,没有精确的单步 oracle,GiGPO 和 PPO 在受限预算下都无法超过 GRPO,基本持平。论文的结论偏悲观:这类监督在多数真实任务里拿不到。\n\n第二条是论文自己的 BudgetMix RL,改动很小:奖励机制不变,只改预算分布。每个任务按 10% 间隔构造候选预算并均匀采样,组内轨迹共享同一预算;训练时把总预算和剩余预算写进 prompt,超预算未完成记零奖励。论文称相对全预算 RL,25%、50%、75% 三档分别提升 7.5、5.3、4.0 个百分点,100% 档持平。\n\n按表 1 的每任务增益行复算,25% 档 6.6 与 8.35 的均值 7.475 对上论文的 7.5,50% 档 5.3125 对 5.3,75% 档 4.0125 对 4.0,三档都能复算。\n\n## 落地前有两道硬门槛\n\n第一道是预算的告知方式。论文所有实验都把总预算与剩余预算写进系统提示,但真实部署里的上限往往是超时或成本天花板,不会事先声明。论文页评论区有读者直接追问:如果上限不被告知,模型学到的是更谨慎的早期策略,还是只在条件化于那个被写明的数字?论文没有回答。\n\n第二道是复现边界。官方仓库自己写明,示例与工程检查并不是论文基准的数值复现,模型权重、私有基准测试、原始轨迹和完整实验结果都没打包。它提供的是一套诊断工具链:紧预算评估、错误前缀评估、训练动态追踪与 BudgetMix 审计。\n\n## 所以呢\n\n如果你的 Agent 上线时有交互轮次上限,评测口径最好加一维 Pass@T,而不是只报任务成功率。更要紧的是,如果任务里混着金融操作、数据库写入这类不可逆动作,单步决策质量才是真正该被考核的东西,而多轮 RL 在这一项上没有保证——它买到的是出错之后爬回来的韧性。\n\n更准确的说法不是「多轮 RL 让 Agent 变聪明了」,而是「多轮 RL 让 Agent 在给足交互预算时更抗造」。\n\n资料来源:论文 https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06483 ,项目页 https:\u002F\u002Frethinking-agentic-rl.github.io\u002F ,官方仓库 https:\u002F\u002Fgithub.com\u002FGaryStack\u002FRethinking-Agent-RL 。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06483","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"c9a50d0f-97a6-4f50-8b45-70468e7a95c4","en","Multi-Turn Agentic RL: What the 46.8% Number Means","Multi-turn agentic RL trajectory gains shrink when turn budgets tighten; single-step quality wins on only 46.8% and 42.8% of matched states.","Multi-turn agentic reinforcement learning has become close to the default recipe for training LLM agents, yet evaluation is still overwhelmingly trajectory-level: did the task get finished. The authors of this paper argue that this habit collapses two very different things into one score — being better at every individual step, and being able to recover after something goes wrong — even though those two carry completely different weight in real deployments.\n\n## A question hidden behind the usual metric\n\nThe work comes from the Key Laboratory of Cognition and Decision Intelligence for Complex Systems at the Institute of Automation, Chinese Academy of Sciences, the School of Artificial Intelligence at the University of Chinese Academy of Sciences, and Tsinghua University. It is public on arXiv, with a project page and an MIT-licensed GitHub repository released alongside.\n\nThe move is to generalize the single-turn Pass@K metric into a two-dimensional Pass@(K,T): K is sampling breadth, meaning how many independently sampled trajectories you get, and T is interaction depth, meaning how many interaction turns each trajectory is allowed. Experiments span four benchmarks — DeepCoder-MT, BFCL, InterCode-SQL, and FrozenLake — using Qwen3-4B and Qwen3-8B as base models and running GRPO, GiGPO, and PPO.\n\n## First counterintuitive result: tighten the budget and the advantage flips\n\nFixing K=8 and sweeping T, the paper’s tables tell a clear story. The RL model’s margin over the base model grows as the interaction budget loosens, but it collapses as the budget tightens, and on some tasks it inverts outright.\n\nThe sharpest cell sits at the 25% budget level of BFCL-v3: the Qwen3-4B base model scores 3.0, while full-budget GRPO RL manages only 2.5 — below the base model. FrozenLake behaves the same way at that tier, with base at 30.3 and full-budget GRPO at 28.7. The authors also compute a structural upper bound for each task at each budget, to rule out the trivial explanation that tight budgets simply make tasks unsolvable. Even with substantial headroom remaining below that upper bound, RL still shows no advantage.\n\n## Second counterintuitive result: single-step actions are not uniformly better\n\nThe authors evaluate at fine granularity on matched states. They sample intermediate states in equal numbers from base and RL trajectories to avoid state-visitation bias, draw G=16 next-step actions from each policy at the same state, and score them by action value — exact Q* values from finite-horizon dynamic programming for FrozenLake with a discount factor of 0.95, and unit-test pass rates for DeepCoder-MT.\n\nThe distribution does not favor RL; it leans slightly toward the base model. RL picks the better single-step decision on only 46.8% of DeepCoder-MT states and 42.8% of FrozenLake states, with average quality no better than the base model and in fact slightly below it.\n\n## But the strength is unmistakable: error recovery\n\nChange the framing and the picture inverts. The authors build an error-state pool by injecting an incorrect action at the preceding step. Here, RL selects the superior action in 65.7% of FrozenLake states and 67.3% of DeepCoder-MT states. Continuing from identical error states, terminal success rates favor RL at 51.3% against 38.6% on FrozenLake, and 38.5% against 22.9% on DeepCoder-MT.\n\nThe BFCL case study is the most engineering-flavored result. They collect 191 trajectory prefixes containing tool-call errors — 91 from RL rollouts, 100 from base — and let both models take one action from each identical prefix. RL’s error correction rate is 41.4% versus 27.2% for the base model. The appendix also notes that RL’s advantage is not evident early in a trajectory and can even trail the base model, only widening as interaction proceeds.\n\n## Mechanism: locally plausible steps, a different policy over whole trajectories\n\nThe likelihood analysis explains why. Evaluating the negative log-likelihood of single-step actions under the base policy gives 12.20 for RL against 10.25 for base, with the two distributions overlapping heavily — RL’s individual actions do not stray outside the base model’s support. But switch to the cumulative log-likelihood ratio over a complete trajectory, and successful RL trajectories reach roughly 228 while successful base trajectories reach only 26.\n\nThe paper then adds a length-matched control, requiring output token counts to differ by at most 10% of the longer sequence: the matched difference is 185.9 nats on Terminal and 805.8 nats on BFCL, with both 95% confidence intervals above zero — 193 pairs across 89 tasks on Terminal, 283 pairs across 146 tasks on BFCL. In other words, the policy shift at the trajectory level is not simply a byproduct of longer outputs.\n\nTraining dynamics nail the point down. On DeepCoder-MT, the trajectory-level reward Rmulti rises by 0.331 while the single-step reward Rsingle drops by 0.097. The authors phrase it as a pattern that grows progressively during training rather than an accident at the final checkpoint.\n\n## Two mitigations, one of them deployable\n\nThe first path is process supervision signals. On FrozenLake, exact Q-value process rewards and GiGPO stabilize performance under tight budgets. On BFCL, where no exact step-level oracle exists, neither GiGPO nor PPO beats GRPO under constrained budgets and simply matches it. The paper’s conclusion here is pessimistic: such supervision is rarely accessible in realistic agent tasks.\n\nThe second is the paper’s own BudgetMix RL, a deliberately small change — the reward mechanism stays the same, only the budget distribution moves. For each task, candidate budgets are constructed at 10% intervals and sampled uniformly, with all trajectories in a group sharing one budget; during training the total and remaining budget are written into the prompt, and any trajectory that exhausts its budget without finishing receives zero reward. The paper reports average gains over full-budget RL of 7.5, 5.3, and 4.0 percentage points at the 25%, 50%, and 75% tiers respectively, while maintaining parity at 100%. Averaging the per-task gain rows by model size yourself reproduces the figures: at the 25% tier, 6.6 and 8.35 average to 7.475, matching the reported 7.5; the 50% tier gives 5.3125 against 5.3, and the 75% tier gives 4.0125 against 4.0 — all three recompute cleanly.\n\n## Two hard gates before you ship this\n\nThe first is how the budget is communicated. Every experiment in the paper puts the total and remaining budget explicitly into the system prompt, but in real deployments the cap is usually a timeout or a cost ceiling that is never announced. A reader on the paper’s page asks this directly: if the cap is not announced, does the model learn a generally more careful early policy, or does it merely condition on the stated number? The paper does not answer.\n\nThe second is the reproduction boundary. The official repository states plainly that its examples and engineering checks are not numerical reproductions of the paper’s benchmarks, and that model weights, private benchmark tests, original traces, and full experiment results are not bundled. What the repo does provide is a diagnostic toolchain: tight-budget evaluation, error-prefix evaluation, training-dynamics tracking, BudgetMix planning and auditing, plus likelihood and process-signal method paths.\n\n## So what\n\nIf your agent has an interaction-turn cap in production, the evaluation metric should grow a Pass@T dimension rather than reporting task success alone — by the paper’s own tables, the same model can flip from ahead to behind simply by moving budget tier. And if the task mixes in irreversible actions like financial operations or database writes, single-step decision quality is the thing that genuinely deserves scrutiny, and multi-turn RL offers no guarantee there. What it buys is resilience: the ability to climb back after a mistake.\n\nSo the accurate phrasing is not \"multi-turn RL makes agents smarter.\" It is closer to \"multi-turn RL makes agents harder to break, given enough interaction budget.\"\n\nSources: paper https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06483 , project page https:\u002F\u002Frethinking-agentic-rl.github.io\u002F , official repository https:\u002F\u002Fgithub.com\u002FGaryStack\u002FRethinking-Agent-RL .","multi-turn-agentic-rl-budget-dependence","2026-11-06T01:18:33Z","2026-11-06T01:18:44.934190Z","2026-11-06T01:18:44.934198Z",true,"agent",40,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"99c1eb7b-9226-4d61-afdb-5d54f920a85c","研究品味实测:AI 智能体的想法匹配只有人类的 18%","ideataste-llm-research-taste-benchmark","2026-11-04T11:13:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"a50b0336-dabe-405e-b8c4-d27156ab6d24","60% 权重转私有测试集:开放权重落后榜首 12 分","artificial-analysis-intelligence-index-v5","2026-10-28T07:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"ba7f04d1-2fe5-4e1d-90a6-94ec3c4b70df","阿里 UserArena:64,729 条购物轨迹当考场","userarena-shopping-agent-user-simulation-benchmark","2026-10-26T09:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"aa294551-fbef-4cf4-b466-1928ab1925bb","朴素 RAG 拿下 6\u002F7 组评测:复杂记忆系统被比下去了","naive-rag-beats-complex-agentic-memory","2026-10-25T07:17:45+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"da6cbb98-3b89-4631-8dbd-59411136fb30","哈佛用 700 家公司拆解 AI 编程智能体:30% 代码增量,49% 合并时延","harvard-coding-agent-review-bottleneck","2026-10-14T11:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"c368ad9f-9308-4a0b-8f5c-3ae4601b48b9","D2K-Bench: 专家设计把 LLM 写 GPU 核提速 33.9%","d2k-bench-llm-gpu-kernel-design-guidance","2026-10-07T03:00:00+00:00"]