[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-mpo-turn-level-credit-long-horizon-agents":3,"topics-all":38,"news-related-13d6adf3-a5c3-46fb-9ff5-a6999dcf0300":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"13d6adf3-a5c3-46fb-9ff5-a6999dcf0300","Critic值轮内近乎恒定:MPO重构Agent信用分配","长程Agent训练默认把信用摊到每个token。新论文实测2048条轨迹:critic值方差的中位89.5%来自轮次之间,MPO据此把信用传播与重要性采样搬到轮级,Qwen3-4B和30B-A3B上九项指标全部压过GRPO、PPO等基线,4B的pass@32到75.90。","训练长程 Agent 的强化学习配方,大多默认把信用摊到 token 级:每生成一个 token 就算一步动作,PPO 系沿着 token 流用 critic 估优势,GRPO 系干脆整条序列共享一个组相对优势。10 月 26 日挂上 arXiv 的论文 MPO(Multiscale Policy Optimization)先做了一件很多人跳过的事——测量,而测出来的数字直接动摇了 token 级信用的默认地位([arXiv:2610.35013](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.35013))。\n\n## 先测量,再设计\n\n团队统计了一个搜索 Agent 训练过程里 8 个 checkpoint 批次的 2,048 条轨迹、16,323 个轮次:critic 值在轮内部相当稳定,大的跳变集中在轮与轮的边界。量化下来,轮均值之间的差异占 critic 值方差的 89.5%(中位数);93.8% 的轨迹里轮间方差大于轮内方差。这组数字的潜台词是:环境真正消费的决策单位是「一轮完整的助手输出」——搜索词、工具调用、最终回答——而不是单个 token。论文的 Figure 2 里,失败轨迹和成功轨迹呈现同样的形态。\n\n## 两条旧路线的缝隙\n\n论文把现有方法归成两族。GRPO、DAPO、GSPO 这类免 critic 的组相对方法,一条 rollout 只派生一个共享优势,分不清哪一轮、哪个 token 更该记功;PPO、VC-PPO、VAPO、SAO 这类 token 级 critic 方法优势估得细,但没有把「轮内生成」与「轮间交互」当成两个尺度。MPO 的立场是两个尺度都要:跨轮传播信用,轮内再做细分。\n\n## MPO 的三个部件\n\n第一,多尺度优势估计 MAE:把整个轮次当宏动作,在轮轴上跑 GAE,信用每交互衰减一次而不是每 token 衰减一次,天然与轮长无关;轮内再用一条截断在本轮内的局部轨迹做修正并中心化——轮内均值被锁死在轮级估计上,β 控制局部修正强度。第二,轮级重要性采样:前向权重与裁剪都用整轮概率比,同时用 stop-gradient 让每个 token 保留自己的策略梯度。第三,critic 直接回归终止回报(λcritic=1,沿用 VAPO 的解耦 GAE)。消融给出的最佳组合是 (λ, λintra, β)=(0.80, 0.95, 1.00)。\n\n## 成绩单:九项全绿\n\n实验用 Qwen3-4B-Thinking 与 Qwen3-30B-A3B-Instruct,在搜索(BrowseComp-Plus、GAIA、HLE 文本子集)、工具使用(Claw-Eval)、自动化(Automation Bench)三类设置下与 GRPO、GSPO、PPO、VAPO、SAO 同预算对比。4B 上 MPO 的 BrowseComp-Plus pass@1 到 48.08,比最强基线 SAO 的 43.75 高 4.33 个百分点;GAIA 39.47(+3.85)、HLE 9.92(+0.66)。30B-A3B 上三项 pass@1 为 50.32 \u002F 43.42 \u002F 10.03,领先 3.59 \u002F 3.80 \u002F 0.12。Claw-Eval 的 pass3 提升 0.95 与 1.12 个百分点;Automation Bench 从 4B 基座的 0.00 拉到 13.33,30B 到 15.00。在论文自己的评测里,九项指标在两个尺寸上全部居首;采样放大后优势仍在,4B 的 pass@32 在 BrowseComp-Plus 冲到 75.90。两个消融最有说服力:把轮级重要性比换回 token 级,4B 搜索三基准 pass@1 分别掉 2.36、3.12、1.03;β=0 砍掉轮内细分,BrowseComp-Plus 掉 6.28,但仍高于 PPO。\n\n## 该泼的冷水\n\n成绩单是论文自测,暂无第三方复现;评测协议是每次训练选最佳 checkpoint、离线评估 5 次取均值,消融每个设置只跑一次。截至发稿,论文没有放出代码或模型,Hugging Face 论文页上没有任何模型、数据集或代码链接。社区评论区也有务实质疑:轮长悬殊时——40 token 的函数调用对 6,000 token 的文件读取——轮级比值的方差如何控制、裁剪是否需要按任务重调,论文没有正面回答。顺带一提,这篇论文在 Hugging Face Daily Papers 榜单上挂在 IQuest 名下,单日冲到第 2(315 票);同门实验室一周前刚开源过 CLI 智能体模型 IQuest-Q1,「先出模型、再出训练方法」的节奏值得盯。这条赛道也不孤独:8 月本站报道过的 AgentOPSD 用递归贝叶斯信念教 GRPO 判断哪个轮次决定胜负,与本文的 critic 结构测量是两条路径,但都指向「轮次级信用」这块新共识地带。\n\n## 所以呢\n\n对训练 Agent 的团队,这篇论文最值钱的可能不是 MPO 本身,而是那个 89.5%:如果你自己的 critic 也是轮内近乎恒定,那 token 级优势的精细感可能大半是噪声,信用分配的颗粒度值得重新审计。评论区已经有人提议把轮级 critic 值拿到推理期当廉价异常信号——单轮值突然偏离就触发重试或人工介入。代码放出来之前,建议把它放进观察名单,而不是直接接进训练管线。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.35013","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"e064d781-0ad6-4264-870f-5b6f3a57e747","en","MPO: agent credit assignment moves from tokens to turns","Turn means carry a median 89.5% of critic variance. MPO moves credit and importance sampling to the turn level, leading all nine metrics.","Reinforcement-learning recipes for long-horizon LLM agents mostly inherit a default from single-turn RL: treat every generated token as an action. PPO-style methods estimate advantages along the token stream with a learned critic; GRPO-style group-relative methods hand one shared advantage to an entire sequence. A paper posted October 26 — Multiscale Policy Optimization (MPO) — starts by measuring whether that default matches how value actually behaves in multi-turn agents, and the measurement challenges it ([arXiv:2610.35013](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.35013)).\n\n## Measure first\n\nThe authors instrumented a search-agent training run: 2,048 trajectories and 16,323 turns across eight checkpoint batches. Critic values stay relatively stable inside a turn and jump at turn boundaries. Quantified, differences between turn means account for a median 89.5% of the critic's token-weighted within-trajectory value variance, and between-turn variance exceeds within-turn variance in 93.8% of trajectories. The implication the paper draws: the environment acts on complete assistant turns — a search query, a tool call, a final answer — so the macro decision unit is the turn, not the token. Failed and successful trajectories in Figure 2 show the same shape.\n\n## The gap between two families\n\nThe paper groups prior methods into two families. Critic-free group-relative methods (GRPO, DAPO, GSPO) derive one shared advantage per rollout and cannot distinguish which turn or token deserved the credit. Token-level critic methods (PPO, VC-PPO, VAPO, SAO) estimate finer advantages but never separate within-turn generation from between-turn interaction. MPO's position is that both scales are needed: propagate credit across turns, then refine it within each turn.\n\n## Three components\n\nFirst, Multiscale Advantage Estimation (MAE): treat each complete turn as a macro action and run GAE on the turn axis, so credit decays once per interaction rather than once per token, independent of turn length; a within-turn local trace, centered, then refines token credit while locking the turn mean, with β scaling the local correction. Second, turn-level importance sampling: forward weighting and clipping use the complete-turn probability ratio, while a stop-gradient construction lets every token keep its own policy gradient. Third, the critic regresses the terminal return directly (λcritic=1, following VAPO's decoupled GAE). The ablated best setting is (λ, λintra, β) = (0.80, 0.95, 1.00).\n\n## The scoreboard\n\nExperiments use Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct across search (BrowseComp-Plus, GAIA, HLE text-only subsets), tool use (Claw-Eval) and automation (Automation Bench), against GRPO, GSPO, PPO, VAPO and SAO under matched budgets. On 4B, BrowseComp-Plus pass@1 reaches 48.08 versus 43.75 for the strongest baseline SAO (+4.33); GAIA 39.47 (+3.85) and HLE 9.92 (+0.66). On 30B-A3B the three pass@1 scores are 50.32 \u002F 43.42 \u002F 10.03, leading by 3.59 \u002F 3.80 \u002F 0.12. Claw-Eval pass3 gains 0.95 and 1.12 points; Automation Bench climbs from a 0.00 base to 13.33 on 4B and 15.00 on 30B. In the paper's own evaluation MPO tops all nine metrics at both scales, and the edge holds as sampling grows: 4B pass@32 hits 75.90 on BrowseComp-Plus. Two ablations carry the argument. Swapping turn-level ratios back to token-level costs 2.36, 3.12 and 1.03 pass@1 on the 4B search benchmarks; removing within-turn allocation (β=0) drops BrowseComp-Plus pass@1 by 6.28 — yet still beats PPO.\n\n## The cold water\n\nAll numbers are self-reported, with no third-party reproduction yet; the evaluation protocol selects the best checkpoint per run and averages five offline evaluations, and each ablation setting has a single run. As of writing, the paper has released no code or models — the Hugging Face paper page links none. Community comments raise a practical concern: with wildly unequal turn lengths, a 40-token function call against a 6,000-token file read, turn-level ratios carry very different variance, and clipping may need per-task retuning; the paper does not answer that directly. The paper sits under the IQuest badge on Hugging Face's Daily Papers at #2 of the day with 315 upvotes; the same group shipped the open-weight CLI agent model IQuest-Q1 a week earlier — model first, training method next, a rhythm worth watching. The lane is also getting crowded: AgentOPSD (August) taught GRPO to spot the decisive turn via recursive Bayesian beliefs — a different mechanism converging on the same turn-level insight.\n\n## So what\n\nThe most transferable asset here may be the 89.5% number, not the algorithm. If your own critic is also near-constant within turns, most of your token-level advantage precision is probably noise — worth auditing before the next training run. One commenter suggests reusing the turn-level critic value at inference as a cheap anomaly score: a sudden deviation within a turn triggers a retry or a human check. Until the code lands, this belongs on the watchlist, not in your training pipeline.","mpo-turn-level-credit-long-horizon-agents","2026-10-28T19:08:42Z","2026-10-28T19:08:58.921130Z","2026-10-28T19:08:58.921151Z",true,"agent",136,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"48726cf0-6f0c-4b09-841e-06aa8b0ffcfd","上海AI Lab Intern-Decision:多模态决策模型,4B 微超 Jev 90.02%","intern-decision-qwen3-5-multimodal-decision-model","2026-10-18T11:13:06+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"206ea36a-1eea-463f-a241-1e3b32f5ec2d","USTC GraphForge:证据图把任务和 rubric 钉在一起,Qwen3.6-27B 涨三基准","graphforge-ustc-qwen36-27b-evidence-graph","2026-10-04T03:05:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"098ef0da-3801-419a-9e33-855d6c6d7ee6","Trinity:递归自我改进,训练基准平均涨 34 分","trinity-capability-driven-recursive-self-improvement","2026-10-28T13:16:03+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"a50b0336-dabe-405e-b8c4-d27156ab6d24","60% 权重转私有测试集:开放权重落后榜首 12 分","artificial-analysis-intelligence-index-v5","2026-10-28T07:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"be62f4f6-6935-4c2c-abe8-9e0732bb04ef","Trajectory2Gym:智能体的成功轨迹不等于好教材","trajectory2gym-compositional-interaction-synthesis","2026-10-27T21:10:28+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"ba7f04d1-2fe5-4e1d-90a6-94ec3c4b70df","阿里 UserArena:64,729 条购物轨迹当考场","userarena-shopping-agent-user-simulation-benchmark","2026-10-26T09:10:00+00:00"]