[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-rl-training-inference-mismatch-dlc":3,"topics-all":38,"news-related-a6cc8527-e099-488d-ad02-9eb997fa4b96":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"a6cc8527-e099-488d-ad02-9eb997fa4b96","On-policy失配拆解:恒定数值差与自放大滞后","LLM 强化学习里,推理引擎采样、训练器算梯度,两套系统让 on-policy 数据悄悄变成 off-policy。论文把每个 token 的误差拆成两块:近似恒定的引擎数值差,与被自身更新反复放大的权重滞后;DLC 修正在有效样本量跌破一半时重同步采样器,把偏差压回两倍噪声底之内。","在大模型的强化学习训练里,「on-policy」这个词的成色可能没你想的足。主流训练栈通常在同一份权重上跑两套系统:推理引擎负责采样回复,训练器负责算梯度,论文里引用的标准配置就是 vLLM 采样加独立训练器。两套系统的数值实现和权重版本对不齐,「从当前策略采样」的假设就悄悄打了折扣——此前已有研究者报告 vLLM 与 FSDP 在完全相同的权重下给出不同的 token 概率。10 月 29 日挂上 arXiv 的[这篇论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40945),把这类训练-推理失配拆到了可测量的粒度。\n\n## 两个来源,两种动力学\n\n作者先做减法:把基建层的干扰对齐消掉——top-p 采样让上报的分布与实际采样的分布不一致、解码后重新切分 token 改变序列、按奖励过滤回复让批次组成依赖结果。对齐之后,真正到达算法层的失配只剩两个:引擎数值差与权重版本差。拆法本身很干净:让训练器在采样器的权重处做估值,每个采样 token 的误差就能精确分成两块——引擎的数值差、训练器的策略滞后。两块的动力学截然不同:数值差在训练全程近似恒定,像一个固定偏置,在每个前缀上做 score centering 就能消掉主要部分;策略滞后则是训练器自己造成的,上一次更新留下的偏差进入下一次更新,又放大下一次滞后,形成正反馈回路。\n\n## 实验设置\n\n训练跑在 verl 里,采样用 vLLM:Qwen3-0.6B 与 Qwen3-1.7B 的 non-thinking 模式,Countdown 任务,±1 奖励,每步 64 个 prompt、各采 8 条回复,SGD,共 40 步,每个配置三个种子。采样器设四档:BF16 加 CUDA graphs 的参考档(on-policy 基准)、FP8 权重(128×128 分块)加动态激活量化加 FP8 KV cache 的量化档、每 64 步才同步一次的 stale 档(40 步内基本停在初始权重),以及两者的组合档。\n\n## 一条失效定律\n\n重要性加权只能在这个反馈环的比值还集中时切断环路。论文给出的判据是滞后比值的有效样本量:跌破一半,修正失效。Qwen3-1.7B 上比值的重尾让交叉点在第 11 步就到来,比高斯预测早三步,重要性采样随即离开噪声底。量级对比很直观:量化源下各修正基本贴着噪声底,策略梯度本身是噪声底的 13 倍,score centering 复现了 on-policy 训练 46.5% 的测试精度;换成滞后源,score centering 冲到 32 倍噪声底、终点测试精度只剩 12.2%,重要性采样加居中为 2.1 倍、精度 32.0%,以每批前训练器为参考的 PPO 则达到 699 倍。\n\n## DLC 与看不见的塌缩\n\n由此导出的修正叫 discrepancy-lag correction:居中数值差、按采样器权重截断滞后比值、在有效样本量跌破一半时把采样器重同步到训练器权重。stale 源下,DLC 把与 on-policy 训练的偏差控制在两倍噪声底之内(Qwen3-0.6B 上 1.9 倍),1.7B 的三条 run 各在第 20、35、36 步触发一次重同步,全程贴住噪声底;给重要性采样配上同阈值重同步,偏差从 115 倍降到 2.6 倍。最扎心的发现是塌缩的隐蔽性:不做重同步时,两次塌缩点落在初始策略概率低于 10^-3 的 token 上的占比冲到约 90%,此前至多 14%;带重同步则全程不超过 11%。可同一时刻,采样器前缀上的 next-token KL 仍落在正常训练的范围里——漂移发生在 stale 采样器到达不了的前缀上,只盯采样器侧的监控,塌缩是看不见的。跨检查点看,这个 off-support 占比对精度下跌的排序相关(Spearman 0.80)也高于偏差本身(0.74)。\n\n## 所以呢\n\n这篇论文的价值不在又一个训练 trick,而在把一个工程痛点变成可测量对象:数值差恒定、滞后自放大,修正手段因此必须分开;有效样本量跌破一半就是重要性加权的失效线,重同步比调系数更根本。对跑异步 RL 的人,监控清单值得加一条——训练器自身响应的 off-support 占比;只在采样器侧量 KL,可能到发散了都还蒙在鼓里。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40945","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"b526c86a-b0ce-4c55-b5f5-494daeef11dd","en","On-Policy Mismatch: Constant Numerics, Self-Amplifying Lag","LLM RL mismatch splits into constant engine numerics and self-amplifying lag; DLC centers, clips and resyncs to stay within twice the noise floor.","On-policy reinforcement learning for large language models may be quietly off-policy. Mainstream training stacks run two systems on the same weights: an inference engine samples the responses and a trainer computes the gradients — the configuration cited in the paper is vLLM sampling plus a separate trainer. When the numerics and weight versions of the two systems disagree, the assumption that training data comes from the current policy silently breaks. Earlier work already reported that vLLM and FSDP assign different token probabilities at identical weights. [This paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40945), posted to arXiv on October 29, cuts the training-inference mismatch down to measurable components.\n\n## Two sources, two dynamics\n\nThe authors first subtract the infrastructure noise: top-p sampling makes the reported distribution differ from the one actually sampled, decoding and re-tokenizing changes the token sequence, and filtering responses by reward makes batch composition depend on the outcome. Once aligned, only two mismatch sources reach the algorithm layer: engine numerics and weight version. The split itself is clean — evaluate the trainer at the sampler's weights, and each sampled-token error divides exactly into a numerical discrepancy of the engine and a policy lag of the trainer. Their dynamics differ completely: the numerical discrepancy stays approximately constant through training, like a fixed bias that per-prefix score centering removes; the policy lag is produced by the trainer's own updates — the bias one update leaves enters the next update and enlarges the next lag, closing a positive feedback loop.\n\n## Experimental setup\n\nTraining runs in verl with vLLM samplers: Qwen3-0.6B and Qwen3-1.7B in non-thinking mode, the Countdown task, rewards of ±1, 64 prompts with 8 samples per step, SGD, 40 steps, three seeds per configuration. Four samplers: a BF16 reference with CUDA graphs (the on-policy baseline), an FP8 sampler (128x128 blocks) with dynamic activation quantization and an FP8 KV cache, a stale sampler that synchronizes only every 64 steps (so within 40 steps it stays at the initial weights), and their combination.\n\n## A failure law\n\nImportance weighting cuts the loop only while the lag ratio stays concentrated. The paper's criterion is the effective sample size of the ratio: once it falls below one half, the correction fails. On Qwen3-1.7B the heavy tail brings the crossing to step 11, three steps before the Gaussian prediction, and importance sampling leaves the noise floor there. The magnitudes are stark: under quantization the corrections stay near the floor while the policy gradient reaches 13 times it, and score centering matches the 46.5% test accuracy of on-policy training; under staleness score centering reaches 32 times the floor and ends at 12.2% accuracy, importance sampling with centering stays at 2.1 times with 32.0%, and PPO referencing the trainer before each batch reaches 699 times.\n\n## DLC and the invisible collapse\n\nThe derived fix is the discrepancy-lag correction: center the numerical discrepancy, clip the lag ratio against the sampler's weights, and resynchronize the sampler when the effective sample size falls below one half. Under a stale sampler, DLC keeps deviation from on-policy training within twice the noise floor (1.9 times on Qwen3-0.6B); on the 1.7B model each of the three runs resynchronizes once, at steps 20, 35 and 36, and stays at the floor throughout. Adding the same-threshold resynchronization to importance sampling lowers its deviation from 115 to 2.6 times the floor. The most striking finding is how invisible the collapses are: without resynchronization, the off-support share — tokens where the initial policy's probability is below 10^-3 — reaches about 90% at the two collapse points, against at most 14% before them; with resynchronization it stays at or below 11% through step 40. Yet at the same time, the next-token KL on the sampler's prefixes remains within the range of healthy runs — the shift starts on prefixes the stale sampler never reaches, so monitoring only the sampler side hides the collapse. Across checkpoints the off-support share ranks the accuracy drop at a Spearman correlation of 0.80, above the 0.74 of the deviation itself.\n\n## So what\n\nThe value of the paper is not another training trick but turning an engineering pain point into a measurable object: constant numerics versus self-amplifying lag demand different fixes; the effective sample size falling below one half is the expiry date of importance weighting, and resynchronization is more fundamental than tuning coefficients. If you run asynchronous RL, add one item to the monitoring checklist — the off-support share of the trainer's own responses. Measure KL only on the sampler side, and you may stay in the dark until the run diverges.","llm-rl-training-inference-mismatch-dlc","2026-11-02T15:15:34Z","2026-11-02T15:15:42.966108Z","2026-11-02T15:15:42.966117Z",true,"agent",128,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"8c579d8d-4114-4d7d-92c1-22f5fab0537b","连续流语言模型跑到 8B：Lafa 单模型自验证解码提速 2.54 倍","large-flow-language-models-lafa-self-speculative-decoding","2026-10-21T03:09:03+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"f2b0bde3-ebb5-49e5-a418-5ece37639d1b","MIPU\u002FMIPI：把 LLM RL 的「训练—推理失配」从工程噪音重写为优化目标","mipu-mipi-rl-mismatch","2026-07-04T08:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"5a4c2a98-6ce7-4e51-8348-118be3083afc","ELDR 把 MoE 推理的「延迟最后一公里」拉直:vLLM 实测 TPOT 最多砍 13.9%","eldr-moe-routing","2026-07-03T12:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"cd82f869-d4a7-45b7-95ce-b66051e9d933","BlockPilot：实例自适应策略学习让扩散式投机解码再下一城,Qwen3-4B 上首破 4.20× 加速","blockpilot-instance-adaptive-block","2026-07-01T14:17:29+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"abf22bbb-eccf-46d0-99d6-debe1596f92b","自验证蒸馏：无需外部教师，LLM如何实现自我进化","self-verified-distill-qwen3-16-7pp-math","2026-05-27T19:00:00+00:00"]