Cinematic video generation is moving from single-shot demos toward multi-shot narrative, but a quieter engineering problem sits underneath: in today's architectures, the multimodal large language model is routinely bypassed by the video generator itself.
The paper states the bypass plainly. A Video DiT carries its own native text-encoder and VAE reference pathways, so as long as those are good enough to reduce the generation loss, the model has no reason to actually read the hidden states the MLLM hands it. The demonstration is stark: under MLLM-latent-only conditioning, Omni-Video 2 (1.3B) outputs noise outright, while the 14B model and UniVideo produce visible artifacts and even split one person's successive actions into two people acting differently.
Three training stages, three separate diseases
FAPL (Factor-Aware Predictive Learning), co-authored by MMLab at HKUST, Tencent Video, and LIX in France, adds no architectural complexity at all. It simply pulls the MLLM back onto the main path with three training stages.
Stage I: bend the generator until it can only use the MLLM. Freeze the MLLM and the connector, switch off the independent text-encoder and VAE reference conditions, and train the DiT with the projected MLLM latents as its only content condition. Sparse observations preserve compatibility with limited-input generation, while dense observations push visual and temporal information down that single path.
Stage II: teach selective editing with paired videos. Character, scene, action, and camera motion are split into four spans of a structured description. Paired videos specify the relationship — the target factor changes, the others are preserved — and during training the whole latent span for the chosen factor is swapped between the pair, after the connector and before the DiT. Spans need not be equal in length; sequence lengths and masks are updated accordingly. The authors are explicit that this does not assume the four factors are already disentangled in representation space.
Stage III: turn around and train the MLLM. The trained DiT is frozen while the MLLM and connector are jointly optimized, with a text-only observation and a text-plus-complete-video observation sharing one target video. The text-only path forces the MLLM to infer temporal information from sparse evidence; the complete-video path keeps reconstruction from degrading. Because the DiT stays fixed, gradients still flow back to the upstream modules without touching the decoder.
The numbers: a win on editing, a loss on motion
Against the UniVideo (13B) base under the same evaluation setup, FAPL lifts Motion Order from 48% to 61%, cuts translation error from 0.0763 to 0.0586 and rotation error from 1.54 to 1.02 radians, raises factor-editing ∩-Acc from 42% to 58%, and moves cross-shot FaceSim-Arc from 0.451 to 0.523. The larger Omni-Video 2 (14B) baseline reaches only 43% and 37% on the first two of those.
One row of the ablation table cuts the other way. Adding factor-span replacement pushes ∩-Acc from 48% to 57%, but Motion Order actually slips from 54% to 53%, and both camera errors worsen slightly. The paper's own reading is that the gain here is concentrated in factor-aware editing and should not be expected to improve temporal control as a side effect.
The training budget is worth noting too: 10K, 15K, and 10K steps for the three stages, AdamW at a learning rate of 1e-5, BF16, over 100K quality-filtered source-target video pairs shared across all stages (25K per factor). Inference uses 50 sampling steps with CFG at 5.0.
So what
The paper closes on the point that targeting the right bottleneck delivers substantial gains without added architectural complexity. That matters because the last two years of video generation have been spent scaling DiTs, sampling steps, and reference pathways — while whether the MLLM is used at all was quietly assumed.
The authors are candid about the ceiling: current MLLMs struggle to interpret 3D blockouts or reason about complex camera motion, and the pretrained DiT's visual fidelity still limits how realistic complex acting can be. The loop where reasoning and generation feed each other has, for now, only an outline.
Paper and full tables: arXiv:2610.30926.