[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-mipu-mipi-rl-mismatch":3,"topics-all":36,"news-related-f2b0bde3-ebb5-49e5-a418-5ece37639d1b":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"f2b0bde3-ebb5-49e5-a418-5ece37639d1b","MIPU\u002FMIPI：把 LLM RL 的「训练—推理失配」从工程噪音重写为优化目标","强化学习已成为大模型后训练标配（GRPO、DPO、PPO 等），但「训练跑着跑着就崩」始终是未解的工程难题。业界长期把根因归到 off-policyness 与 KL 漂移，然后在损失函数里加熵正则、加 clip、加 reference policy——可崩还是崩。\n\n2026 年 6 月 28 日挂 arXiv 的论文《The Mirage of Optimizing Training Policies》（Liang 等人，含腾讯体系研究者 Bo Zheng）给了一个不一样的解读：**现有的所有优化工作，都被一个被忽略的「目标错位」污染了**。\n\nLLM RL 在 rollout 阶段用推理引擎，在 policy update 阶段用训练引擎；两个引擎即便权重同步，对同一条轨迹也会给出不一致的 token 概率，这叫「训练—推理失配（TIM）」。先前工作把 TIM 当成 off-policyness 噪声去压制，本文则指出 TIM 是一种独立的结构性偏差：训练引擎里「看起来有效」的更新，部署到推理引擎上未必真的变好。\n\n基于此，作者提出两个组件：\n\n- **Monotonic Inference Policy Improvement (MIPI)** —— 显式把「让部署用的推理策略单调变好」写进优化目标；\n- **Monotonic Inference Policy Update (MIPU)** —— 两步框架：先用采样器构造候选更新，再用 inference 侧的 gap proxy 选择性接受同步候选。\n\n论文在两个模型规模、高失配条件下实测，MIPU 在平均推理性能与训练稳定性上都取得可观测提升，并减少训练崩溃频次。\n\n**我的看法**：这件事的意义不在那点指标，而在把 RL 后训练工程化拉到「第一性原则」层面。所有做 GRPO \u002F RLVR \u002F Online DPO 的团队，建议重新审视「训练—推理一致性」——它可能比训出来的策略本身更影响最终交付质量。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.29526","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"e1a1683d-c0e5-4f26-826a-b3e42e8d21bd","en","MIPU\u002FMIPI turns RL train-inference mismatch into an objective","Reinforcement learning has become the standard for large-model post-training (GRPO, DPO, PPO, etc.), but \"training runs and then crashes\" is still an unsolved engineering problem. The industry has long attributed the root cause to off-policyness and KL drift, then added entropy regularization, clip, and reference policy in the loss function — but it still crashes. The paper \"The Mirage of Optimizing Training Policies\" (Liang et al., including Tencent researcher Bo Zheng), posted to arXiv on June 28, 2026, gives a different interpretation: **all existing optimization work has been contaminated by a neglected \"objective misalignment\"**. LLM RL uses the inference engine during the rollout phase, and the training engine during the policy update phase; even if the two engines' weights are synchronized, they give inconsistent token probabilities for the same trajectory, this is called \"Training-Inference Mismatch (TIM)\". Previous work treats TIM as off-policyness noise to suppress, this paper instead points out that TIM is an independent structural bias: an update that \"looks effective\" in the training engine may not actually be better when deployed to the inference engine. Based on this, the authors propose two components: - **Monotonic Inference Policy Improvement (MIPI)** — explicitly writing \"let the deployed inference policy monotonically improve\" into the optimization objective; - **Monotonic Inference Policy Update (MIPU)** — a two-step framework: first use a sampler to construct candidate updates, then use the inference-side gap proxy to selectively accept sync candidates. The paper is measured on two model scales under high-mismatch conditions, MIPU achieves observable improvement in both average inference performance and training stability, and reduces the frequency of training crashes. **My view**: the significance of this isn't those few metrics, but pulling RL post-training engineering to the \"first-principles\" level. All teams doing GRPO \u002F RLVR \u002F Online DPO are advised to re-examine \"training-inference consistency\" — it may affect the final delivered quality more than the trained policy itself.","mipu-mipi-rl-mismatch","2026-07-04T08:00:00Z","2026-07-04T08:07:13.019947Z","2026-08-19T02:08:40.142862Z",true,"agent",262,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"5a4c2a98-6ce7-4e51-8348-118be3083afc","ELDR 把 MoE 推理的「延迟最后一公里」拉直:vLLM 实测 TPOT 最多砍 13.9%","eldr-moe-routing","2026-07-03T12:30:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"cd82f869-d4a7-45b7-95ce-b66051e9d933","BlockPilot：实例自适应策略学习让扩散式投机解码再下一城,Qwen3-4B 上首破 4.20× 加速","blockpilot-instance-adaptive-block","2026-07-01T14:17:29+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"abf22bbb-eccf-46d0-99d6-debe1596f92b","自验证蒸馏：无需外部教师，LLM如何实现自我进化","self-verified-distill-qwen3-16-7pp-math","2026-05-27T19:00:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"6c9e553d-3029-4867-bc29-ee069d26b934","自验证蒸馏：无需外部教师，LLM 如何实现自我进化","svd-self-verified-distill-stanford-perplexity","2026-05-27T16:05:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"2ca620f4-a046-4274-925d-f0689123a498","ParaRNN：Apple 让 RNN 重回战场，7B 参数模型训练提速 665 倍","apple-pararnn-7b-rnn-665x-faster-iclr","2026-05-27T13:15:00+00:00"]