[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-fapl-mllm-latents-cinematic-video-generation":3,"topics-all":38,"news-related-dc534f1a-dbe2-4f79-9557-bbb43a82f24f":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"dc534f1a-dbe2-4f79-9557-bbb43a82f24f","视频模型总在绕过 MLLM,三段训练把它拉回来","港科大 MMLab、腾讯视频与 LIX 联合署名的 FAPL 指出:视频 DiT 会绕过多模态大模型自带的文本与 VAE 通道,单用 MLLM latent 反而出噪声。三段训练不动架构,靠冻结解冻交替把隐藏状态拉回主干,Motion Order 由 48% 升到 61%,旋转误差降到 1.02。","电影感视频生成正从单镜头炫技走向多镜头叙事，但底下藏着一个更隐蔽的问题：现在的架构里，多模态大模型（MLLM）经常被视频生成模型直接绕过。\n\n论文的观察很直接：DiT 自带原生文本编码器和 VAE 参考图通道，只要这些通道够用，模型就能自己把生成损失降下去，不必认真读 MLLM 的隐藏状态。实测最能说明问题——把条件换成只用 MLLM latent 时，Omni-Video 2 (1.3B) 直接吐噪声，14B 和 UniVideo 则产出明显伪影，甚至把同一个人连续动作演成两个人各干各的。\n\n## 三段训练，各自治一个病\n\n港科大 MMLab、腾讯视频与法国 LIX 联合署名的 FAPL（Factor-Aware Predictive Learning）不加任何架构改动，只用三段训练把 MLLM 拉回主干道。\n\n**Stage I：先把生成器掰到只能用 MLLM。** 冻结 MLLM 和 connector，关掉独立的文本编码器与 VAE 参考通道，把 MLLM 投影后的 latent 当作唯一内容条件训练 DiT。稀疏观测保持有限输入也能生成的兼容性，稠密观测把视觉与时序信息压进这条通路。\n\n**Stage II：用成对视频教「只改该改的」。** 角色、场景、动作、镜头运动被拆成结构化描述里的四段文本。成对视频给出「目标因子要变、其余要保」的关系，训练时在 connector 之后、DiT 之前，把配对样本中对应因子的整段 latent 换掉（不要求等长，mask 同步更新），再让 DiT 对着目标视频学习。作者说明，这不假设四种因子已解耦。\n\n**Stage III：反过来训 MLLM。** 冻结训好的 DiT，联合优化 MLLM 和 connector，用「纯文本」和「文本加完整视频」两种观测共享同一目标视频。纯文本一路逼 MLLM 从稀疏证据推断时序，完整视频一路保证重建不退化；因为 DiT 固定，梯度仍穿回上游更新，无需同步改解码器。\n\n## 数字：赢在编辑，输在动作\n\n同口径下 FAPL 相对基座 UniVideo (13B)：Motion Order 从 48% 提到 61%，平移误差 0.0763 降到 0.0586，旋转误差 1.54 降到 1.02 弧度，因子编辑 ∩-Acc 从 42% 提到 58%，跨镜头 FaceSim-Arc 从 0.451 升到 0.523。参数更大的 Omni-Video 2 (14B) 只有 43% 与 37%。\n\n消融表有一行反直觉：加入因子段替换后，∩-Acc 从 48% 跳到 57%，但 Motion Order 反而从 54% 微降到 53%，两项镜头误差也略微变差。论文的解释是，这段收益集中在因子编辑上，不指望顺带改善时序。\n\n训练成本：三段分别 10K、15K、10K 步，AdamW 1e-5，BF16，共享 100K 条质量过滤后的视频对（每因子 25K）；推理 50 步采样，CFG 5.0。\n\n## 所以呢\n\n收尾句是：找对瓶颈，能带来可观增益而不需要增加架构复杂度。分量在于——视频生成这两年堆 DiT 规模、采样步数、参考图通道，而 MLLM 有没有被真正用上，此前一直被当成理所当然。\n\n作者也承认边界：当前 MLLM 读不懂 3D blockout、推不明白复杂运镜，预训练 DiT 的视觉保真度仍限死复杂表演。推理与生成互相喂养的闭环，目前只见轮廓。\n\n论文与完整表格见 [arXiv:2610.30926](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.30926)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.30926","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"ebe5dcd1-46b1-4298-b8c2-8e0e2f456e56","video-generation",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"e3853cd5-7da6-41cd-9c8e-8f24eb8ac9da","en","Video Models Keep Bypassing the MLLM","A paper co-authored by MMLab at HKUST, Tencent Video, and LIX shows that video DiTs routinely bypass the multimodal large language model entirely, relying on their own text and VAE pathways so that MLLM latents alone yield noise. Three training stages with no architectural change pull the hidden states back onto the main path, lifting Motion Order from 48% to 61% and cutting rotation error to 1.02 radians.","Cinematic video generation is moving from single-shot demos toward multi-shot narrative, but a quieter engineering problem sits underneath: in today's architectures, the multimodal large language model is routinely bypassed by the video generator itself.\n\nThe paper states the bypass plainly. A Video DiT carries its own native text-encoder and VAE reference pathways, so as long as those are good enough to reduce the generation loss, the model has no reason to actually read the hidden states the MLLM hands it. The demonstration is stark: under MLLM-latent-only conditioning, Omni-Video 2 (1.3B) outputs noise outright, while the 14B model and UniVideo produce visible artifacts and even split one person's successive actions into two people acting differently.\n\n## Three training stages, three separate diseases\n\nFAPL (Factor-Aware Predictive Learning), co-authored by MMLab at HKUST, Tencent Video, and LIX in France, adds no architectural complexity at all. It simply pulls the MLLM back onto the main path with three training stages.\n\n**Stage I: bend the generator until it can only use the MLLM.** Freeze the MLLM and the connector, switch off the independent text-encoder and VAE reference conditions, and train the DiT with the projected MLLM latents as its only content condition. Sparse observations preserve compatibility with limited-input generation, while dense observations push visual and temporal information down that single path.\n\n**Stage II: teach selective editing with paired videos.** Character, scene, action, and camera motion are split into four spans of a structured description. Paired videos specify the relationship — the target factor changes, the others are preserved — and during training the whole latent span for the chosen factor is swapped between the pair, after the connector and before the DiT. Spans need not be equal in length; sequence lengths and masks are updated accordingly. The authors are explicit that this does not assume the four factors are already disentangled in representation space.\n\n**Stage III: turn around and train the MLLM.** The trained DiT is frozen while the MLLM and connector are jointly optimized, with a text-only observation and a text-plus-complete-video observation sharing one target video. The text-only path forces the MLLM to infer temporal information from sparse evidence; the complete-video path keeps reconstruction from degrading. Because the DiT stays fixed, gradients still flow back to the upstream modules without touching the decoder.\n\n## The numbers: a win on editing, a loss on motion\n\nAgainst the UniVideo (13B) base under the same evaluation setup, FAPL lifts Motion Order from 48% to 61%, cuts translation error from 0.0763 to 0.0586 and rotation error from 1.54 to 1.02 radians, raises factor-editing ∩-Acc from 42% to 58%, and moves cross-shot FaceSim-Arc from 0.451 to 0.523. The larger Omni-Video 2 (14B) baseline reaches only 43% and 37% on the first two of those.\n\nOne row of the ablation table cuts the other way. Adding factor-span replacement pushes ∩-Acc from 48% to 57%, but Motion Order actually slips from 54% to 53%, and both camera errors worsen slightly. The paper's own reading is that the gain here is concentrated in factor-aware editing and should not be expected to improve temporal control as a side effect.\n\nThe training budget is worth noting too: 10K, 15K, and 10K steps for the three stages, AdamW at a learning rate of 1e-5, BF16, over 100K quality-filtered source-target video pairs shared across all stages (25K per factor). Inference uses 50 sampling steps with CFG at 5.0.\n\n## So what\n\nThe paper closes on the point that targeting the right bottleneck delivers substantial gains without added architectural complexity. That matters because the last two years of video generation have been spent scaling DiTs, sampling steps, and reference pathways — while whether the MLLM is used at all was quietly assumed.\n\nThe authors are candid about the ceiling: current MLLMs struggle to interpret 3D blockouts or reason about complex camera motion, and the pretrained DiT's visual fidelity still limits how realistic complex acting can be. The loop where reasoning and generation feed each other has, for now, only an outline.\n\nPaper and full tables: [arXiv:2610.30926](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.30926).","fapl-mllm-latents-cinematic-video-generation","2026-10-23T09:20:00Z","2026-10-23T09:15:04.648929Z","2026-10-23T09:15:04.648937Z",true,"agent",26,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"384bb931-49cb-4b1c-b1a8-89495a166699","腾讯ARC Lab:自回归视频的捷径问题,FCP零成本修补","tencent-arc-autoregressive-video-fcp-future-chunk-prediction","2026-10-23T05:05:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"cf7aeaaa-7179-4de2-b177-9adf54668d61","StreamEval：37 模型实测，实时视频最缺听指令","streameval-interactive-streaming-video-benchmark","2026-10-22T11:08:09+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"63299a5b-f3b7-40c4-9985-0132e0cbffac","多轮图像编辑漂移的解法：MT Forcing 用自回归扩散把单轮质量带进多轮","mt-forcing-multi-turn-image-editing","2026-10-20T05:06:17+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"abd2ad4a-d732-41f0-b1a1-5711103ef7a6","Orbia 基准:生成视频还构不成一致的 3D 世界","orbia-3d-consistency-benchmark","2026-10-18T15:12:16+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"d0d8e772-57f7-47a0-8fd8-dce09f3bd1e5","浙大ATAS登顶HF论文榜:给AI视频装上持久世界记忆","atas-world-state-agentic-video-generation","2026-10-15T15:15:45+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"240d8a4c-c025-4080-be14-06099f650a71","xAI 把视频价格打到底:1080p 蒸馏版 Lite 比 Veo 3.1 便宜 65%","grok-imagine-1-5-lite-cheaper-than-veo","2026-10-10T08:00:00+00:00"]