arXiv 2605.22344 introduces Bernini, an open-source video generation framework from ByteDance that uses an MLLM (Multimodal Large Language Model) as a "semantic planner" to separate the "thinking" from the "rendering" in video generation. The standout: the semantic planner decides "what should happen in the video," and a separate rendering model generates the actual pixels.
The "semantic planner + renderer" insight: traditional video generation models do both "thinking" and "rendering" in a single model. The "thinking" — deciding what objects to show, what actions to take, what the narrative is — is conceptually different from "rendering" — converting the decision into pixels. Bernini's fix: separate the two, with a "semantic planner" (an MLLM) handling the "thinking" and a "renderer" (a video diffusion model) handling the "rendering."
The technical details: the semantic planner takes a text prompt and outputs a structured "scene description" — a sequence of (object, action, time) triples that describe the video. The renderer takes the scene description and generates the video. The two are trained jointly, with the semantic planner's output being a "soft constraint" on the renderer's generation.
The benchmark: Bernini-generated videos score higher on "narrative consistency" and "object persistence" than single-model video generation. The "semantic planner" gives the model a higher-level understanding of the video, and the "renderer" focuses on visual quality.
The bigger takeaway: "semantic planning" is a significant new direction for video generation. The "one model does everything" approach has hit a quality ceiling, and the "semantic planner + renderer" approach is a clean way to break through. For the industry, this signals that the next round of video generation models will adopt the "planner + renderer" architecture, and the "best video model" will be the one with the best semantic planner.