[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-outcome-mass-alignment-multi-turn-agent-rl":3,"topics-all":38,"news-related-a2038f05-8ad2-4b57-bb29-47ff522e09a0":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"a2038f05-8ad2-4b57-bb29-47ff522e09a0","失败轨迹偷走权重:ALFWorld成功率涨7到9.4点","多轮智能体的强化学习里,同一条轨迹会被拆成多行多token,失败轨迹因步数更长反而占掉大部分训练权重。论文提出OMA:按唯一源轨迹统计结果、按轨迹分配目标质量,在ALFWorld上把成功率从73.4%提到82.8%,而在普通WebShop上甚至会下降。","在 ALFWorld 这类多轮家务环境里训练智能体，奖励是稀疏的 episode 级信号：可能要走十几步才拿回一个 0 或 1。论文作者举了个具体例子——同一个任务，策略采样出两条轨迹，一条 14 轮成功、1,663 个 response token，另一条 50 轮失败、6,446 个 token。按论文的算法，失败那条在展开后占掉了 78.1% 的结果人口和 79.5% 的目标质量。论文是《Outcome–Mass Alignment》（arXiv:2611.16816，https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16816）。\n\n## 一条轨迹在训练机里不是一个单位\n\n多轮智能体的强化学习有两种把交互喂给优化器的做法。轨迹级优化（RAGEN）把整条交互拼成一条序列，上下文随轮数膨胀；分步优化把每一轮切成独立的 response 行，单行上下文可控，代价是同一条源轨迹被摊成数量不定的行和 token。GiGPO、HGPO 走的是后一条路线，论文盯的也是它。\n\n分步展开带来两个层次完全不同的问题。第一个在结果统计层：同一个 episode 的结果被贴到它展开的每一行，按行做组内统计时这个结果被数了 n_i 次。论文给了个极简算例：一条成功轨迹（y=1，n=1）和一条失败轨迹（y=0，n=3），按轨迹算均值是 0.5，按行算变成 0.25——组内基线被失败轨迹的展开倍数拖低了。\n\n第二个在目标质量层：token 平均的损失里，一条轨迹拿到的系数是 D_i 除以全组 D 的和（D_i 是它的有效 token 数），行平均则是 n_i 除以全组 n 的和。也就是说，一条失败的轨迹因为跑得更久，天然在梯度里占更大的位置。\n\n## OMA 只动两处\n\n论文的方案叫 Outcome–Mass Alignment，两个组件可以单开也可以一起开。Outcome Alignment 把结果统计挪回唯一的源轨迹上算，再把算好的优势广播回它的行——行被复制多少次都不影响统计量。Mass Alignment 先在轨迹内部聚合 token 贡献，再让每条源轨迹拿到相等的外部质量（完整组里是 1\u002FK），与它展开成多少行、多少 token 无关。\n\n关键是 OMA 不碰优化器原生的东西：轨迹内的信用分配、importance ratio、clipping 全部原样保留。它只是把「一条轨迹该占多大影响力」从隐式变成显式。GRPO、GiGPO、GSPO 都能接。\n\n## 涨在哪，以及在哪里失效\n\nALFWorld 上跑 GRPO 基线，底座是 Qwen2.5-1.5B\u002F3B-Instruct，前 150 次更新、每组 8 条轨迹、batch 16、4 张卡、种子 0。OMA 把 1.5B 的成功率从 73.4% 提到 82.8%，3B 从 74.2% 提到 81.2%，摘要里的说法是同预算终点成功率提升 7.0 到 9.4 个百分点。\n\n消融更能说明问题，两个组件不通用。3B 上基线 74.2%，只开 Outcome Alignment 反而掉到 61.7%，只开 Mass Alignment 是 77.3%，两个都开才是 81.2%；1.5B 上更极端，单独 OA 只有 38.3%，比基线 73.4% 差一大截，单独 MA 是 76.6%，OMA 82.8%。\n\n边界出现在 WebShop。普通 WebShop 上 1.5B 终点成功率 65.6% 升到 70.3%、任务分 76.2% 升到 78.8%，但成功率 nAUC 从 39.1% 只走到 39.0%，任务分 nAUC 从 63.3% 掉到 57.9%；3B 直接反转，成功率 64.8% 掉到 57.0%，任务分 81.4% 掉到 71.4%。论文的解释是普通 WebShop 里「长轨迹」和「失败」的耦合弱甚至反向，这时候改记账方式就没收益。改造成成功导向的 WebShop-SOL 后耦合被拉回来，成功率 nAUC 从 39.9% 提到 51.1%，终点成功率 64.8% 提到 73.4%。\n\n换优化器也成立：GiGPO 终点成功率 88.3% 到 89.8%、nAUC 63.1% 到 67.3%；GSPO 是 65.6% 到 70.3%、nAUC 42.8% 到 49.2%。\n\n## 两个修正在往相反方向拉\n\n论文用冻结批次的梯度审计看这件事：重算同一批轨迹在四种口径下的梯度。WebShop 那个近似对称的批次里，OA 和 MA 的余弦相似度是 0.996；ALFWorld 上展开越不对称，这个值掉到 -0.421、-0.506、-0.368，GiGPO 那行是 -0.667。负值意味着两个修正在往相反方向推，不是一个标量缩放。在失败与成功的 token 比达到 8.78 的那批上，OMA 把梯度范数比从 5.05 压到 1.60，接近原始口径。\n\n## 另一支团队独立撞上同一个机制\n\n「失败轨迹系统性更长」不是只有这篇在看。AsyncWebRL（arXiv:2606.05597，https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.05597）在视觉网页智能体的多步 GRPO 上独立诊断出：每条轨迹的 1\u002F|τ_i| 归一化会压低失败 token 上的负梯度，策略于是持续产出啰嗦的输出结构，作者把 1\u002F|τ_i| 换成常数 1\u002Fk 来切断这个耦合。OMA 把 AsyncWebRL 列为最接近的对比工作，同时指出两者外层加权策略不同：OMA 不偏好短轨迹，也不让质量随 H_i 缩放。\n\nGiGPO（arXiv:2505.10978，https:\u002F\u002Farxiv.org\u002Fabs\u002F2505.10978）是另一条路线，不改外层记账，而是造 anchor state 分组、在轨迹内部做细粒度信用分配，论文自报在 ALFWorld 和 WebShop 上分别比 GRPO 高出 12% 和 9% 以上。三条路线拼起来看，多轮 Agent 的 RL 正在从「只调优化器」转向「连外层记账一起调」。\n\n## 论文自己划的边界\n\n几个限制得一起记着。主实验只有一个种子、两个环境、Qwen2.5 的 1.5B 和 3B 两档。附录补的两种子实验里，ALFWorld-3B 平均终点成功率从 70.31% 提到 86.33%，但 WebShop-1.5B 的 nAUC 是往回走的（成功率 nAUC 从 30.29% 到 29.15%，任务分 nAUC 从 45.37% 到 43.89%），论文说这些是描述性数字，没有做显著性检验。对齐更新步数不等于对齐生成的 token 数和墙钟时间，不能拿它比效率。论文还交代 WebShop 与 WebShop-SOL 的对照同时改了模型档位和终止条件，不构成隔离的因果检验。\n\n## 所以呢\n\n这篇论文的价值不在那几个百分点，而在它把一个一直默认存在的记账动作拆开看：多轮 RL 里，一条轨迹在优化器眼里到底算几个样本，从来不是优化器定的，是训练管线的展开方式顺手定的。展开方式一变，失败轨迹的权重就跟着变，而失败轨迹通常是最长的那批。\n\n所以下次看到某个多轮智能体训练配方报出漂亮曲线，值得多问一句：它按行算还是按轨迹算组内结果？它的外部质量是按 token 摊的还是按轨迹摊的？这两个问题的答案，可能比再多调几个超参更影响结果。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16816","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"691d0e22-2fae-40f8-a523-7bb85f66311f","en","Failed Trajectories Hijack Multi-Turn RL Weight","Stepwise training expands one trajectory into many rows, so long failures dominate the signal. OMA lifts ALFWorld success from 73.4% to 82.8%.","In multi-turn household environments such as ALFWorld, training an agent means dealing with sparse episode-level reward: the policy may take a dozen steps before earning back a single 0 or 1. The paper opens with a concrete pair. On one task, the same checkpoint samples two trajectories, one a 14-turn success with 1,663 response tokens, the other a 50-turn failure with 6,446 tokens. Under the paper's accounting, that failure ends up representing 78.1% of the row-expanded outcome population and 79.5% of the token-weighted objective mass. The paper is Outcome-Mass Alignment (arXiv:2611.16816, https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16816).\n\n## One trajectory is not one unit inside the trainer\n\nThere are two ways to hand a multi-turn interaction to the optimizer. Trajectory-wise optimization (RAGEN) concatenates the whole interaction into one sequence, so the effective context grows with the horizon. Stepwise optimization instead materializes each turn as a separate response row conditioned on a managed history, which keeps every individual context tractable but expands one source trajectory into a variable number of rows and tokens. GiGPO and HGPO follow the stepwise route, and so does this paper's analysis.\n\nThat expansion creates two problems at different levels. The first lives in the outcome statistics. The same episode outcome is attached to every row the trajectory produced, so row-based group statistics count it n_i times. The paper gives a minimal example: one successful trajectory (y=1, n=1) and one failed trajectory (y=0, n=3). Averaged over source trajectories the group mean is 0.5; averaged over expanded rows it becomes 0.25. The in-group baseline has been dragged down by how many rows the failure happened to expand into.\n\nThe second problem lives in objective mass. Under a token-mean loss, a trajectory receives a coefficient of D_i over the sum of D across the group, where D_i is its valid-token count. Under a row-mean loss it receives n_i over the sum of n. In other words, a failed trajectory, purely because it ran longer, naturally occupies more of the gradient.\n\n## OMA changes exactly two things\n\nThe method is called Outcome-Mass Alignment, and its two components can be enabled separately or together. Outcome Alignment computes the outcome statistics over unique source trajectories and then broadcasts the resulting advantage back to that trajectory's rows, so duplicating rows can no longer shift the statistic. Mass Alignment first aggregates token contributions within each trajectory and then gives every source trajectory the same outer mass, 1\u002FK for a complete group, regardless of how many rows or tokens it expanded into.\n\nWhat OMA deliberately leaves alone matters as much as what it changes: within-trajectory credit assignment, importance ratios, and clipping all stay native to the base optimizer. It only turns the question of how much influence one trajectory should carry from an implicit by-product into an explicit choice. GRPO, GiGPO, and GSPO can all host it.\n\n## Where it helps, and where it stops helping\n\nThe ALFWorld baseline is GRPO on Qwen2.5-1.5B\u002F3B-Instruct, using the first 150 trainer updates, eight trajectories per task group, batch size 16, one PPO epoch, seed 0, and four GPUs. OMA lifts 1.5B success from 73.4% to 82.8% and 3B from 74.2% to 81.2%; the abstract phrases this as a 7.0 to 9.4 percentage-point gain in matched-horizon endpoint success.\n\nThe ablations make the point sharper, because the two components are not interchangeable. At 3B the baseline sits at 74.2%, Outcome Alignment alone actually drops to 61.7%, Mass Alignment alone reaches 77.3%, and only the combination gets to 81.2%. At 1.5B the split is more extreme: OA on its own yields just 38.3%, a long way below the 73.4% baseline, while MA alone gives 76.6% and full OMA gives 82.8%.\n\nThe boundary shows up on WebShop. On ordinary WebShop at 1.5B, endpoint success improves from 65.6% to 70.3% and task score from 76.2% to 78.8%, but success nAUC only moves from 39.1% to 39.0% and score nAUC falls from 63.3% to 57.9%. At 3B it reverses outright: success drops from 64.8% to 57.0% and score from 81.4% to 71.4%. The paper's explanation is that ordinary WebShop has weak or even reversed coupling between long trajectories and failure, so changing the accounting buys nothing. In a success-oriented variant, WebShop-SOL, that coupling is restored: success nAUC rises from 39.9% to 51.1% and endpoint success from 64.8% to 73.4%.\n\nThe result also survives a different optimizer. GiGPO goes from 88.3% to 89.8% endpoint success with nAUC from 63.1% to 67.3%; GSPO goes from 65.6% to 70.3% with nAUC from 42.8% to 49.2%.\n\n## The two corrections pull in opposite directions\n\nThe paper inspects this through a frozen-batch gradient audit, recomputing the gradients of the same batch under four accounting regimes. In the near-symmetric WebShop batch the cosine between OA and MA is 0.996. On ALFWorld, as expansion asymmetry grows, that value falls to -0.421, -0.506, and -0.368, with the GiGPO row at -0.667. Negative values mean the two corrections act in opposing directions rather than as a scalar rescaling. In the batch with the strongest asymmetry, where the failed-to-successful token ratio is 8.78, OMA pulls the gradient norm ratio down from 5.05 to 1.60, close to the raw regime.\n\n## Another team hit the same mechanism independently\n\nSystematically longer failure trajectories are not something only this paper noticed. AsyncWebRL (arXiv:2606.05597, https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.05597) independently diagnoses, for multi-step GRPO on visual web agents, that the per-trajectory normalizer 1\u002F|tau_i| down-weights the negative gradient on failed tokens, so the policy keeps producing verbose output structures; the authors break that coupling by replacing 1\u002F|tau_i| with the constant 1\u002Fk. OMA names AsyncWebRL as its closest comparison on trajectory-length weighting while noting that the two choose different outer-weighting policies: OMA neither prefers shorter trajectories nor scales mass with H_i.\n\nGiGPO (arXiv:2505.10978, https:\u002F\u002Farxiv.org\u002Fabs\u002F2505.10978) takes a third route. Instead of touching outer accounting it builds anchor state groups for fine-grained credit assignment inside a trajectory, and the paper reports gains of more than 12% on ALFWorld and more than 9% on WebShop over GRPO. Taken together the three lines of work point at a shift in how multi-turn agent RL is being done, from tuning the optimizer alone to tuning the outer accounting as well.\n\n## The boundaries the paper draws itself\n\nA few caveats need to travel with the numbers. The main-text experiments use a single seed, two environments, and only the 1.5B and 3B Qwen2.5 sizes. In the appendix two-seed runs, ALFWorld-3B mean endpoint success rises from 70.31% to 86.33%, but WebShop-1.5B nAUC moves the other way, with success nAUC from 30.29% to 29.15% and score nAUC from 45.37% to 43.89%; the paper calls these descriptive and reports no significance test. Matching trainer updates does not equalize generated tokens, environment interactions, or wall-clock time, so the comparison cannot be read as an efficiency result. The paper also states plainly that the WebShop versus WebShop-SOL contrast changes model scale and termination at the same time, so it is not an isolated causal test.\n\n## So what\n\nThe value of this paper is not the few percentage points. It is that it takes an accounting decision that has always been made silently and takes it apart for you. In multi-turn RL, how many samples a trajectory counts as in the optimizer's eyes was never the optimizer's decision; it was a side effect of how the training pipeline materialized the trajectory. Change the materialization and the weight on failure trajectories changes with it, and failed trajectories are usually the long ones.\n\nSo the next time a multi-turn agent training recipe reports a beautiful curve, it is worth asking two more questions: does it compute group statistics over rows or over source trajectories, and is its outer mass spread by token or by trajectory? The answers to those two questions may matter more than another sweep of hyperparameters.\n","outcome-mass-alignment-multi-turn-agent-rl","2026-11-14T11:20:00Z","2026-11-14T11:06:38.747058Z","2026-11-14T11:06:38.747072Z",true,"agent",74,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"6c8ae96a-53e0-4ef7-8609-40b345cdee7c","决策模型实测:自己演砸42.9%,教学生82.9%","decision-model-actor-judge-teacher","2026-11-15T21:45:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"bd350599-7253-4f32-a874-9224bab89a49","多智能体搜索实测：32组对照里委派多数时候反而更差","multi-agent-search-delegation-accuracy","2026-11-15T09:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"9531b4f7-3f4d-474c-b233-c715925a3e76","8.4B轨迹压过25.3B技能语料:形式化证明中训配方","davinci-prover-agent-native-mid-training","2026-11-14T07:20:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"343f6cae-ae14-4617-a317-7316748e1d6e","删掉推理链再转发:同档32B模型差51分","deletion-sensitivity-reasoning-trace-routing","2026-11-13T07:05:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"365a0af2-d564-4eff-bf03-a138cdea0c5f","研究智能体大考:只查交付过69%,查全要求剩28%","are-bench-research-agent-context-requirements","2026-11-13T01:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"f2723003-43ac-47fe-bb59-49030446919a","28个决策模型大考:多选最高才36.81%","mmdm-decision-model-benchmark","2026-11-12T13:30:00+00:00"]