On November 3, the Kling team published a technical report on arXiv, "Behind Video Generation," laying out the multimodal understanding stack behind four generations of Kling video models. The central thesis: generation quality is the projection of multimodal understanding through the entire data-conditioning-alignment-evaluation chain, and the bottleneck is shifting from rendering to semantic control.
The unit of progress changed from frame to sequence
The report traces four years: 2023 was still aspirational, 2024 brought compelling but unreliable demonstrations, 2025 delivered production-usable shots, and 2026 frontier systems generate coherent short-form sequences that coordinate multiple shots, persistent subjects, performance, camera language, dialogue, music, and sound. The unit of progress moved from frame to clip, to shot, to the complete sequence. Sequence-level quality is compositional: constraints interact and errors accumulate, so the system must infer a latent creative specification from heterogeneous inputs and express it as conditions the generator can execute.
One backbone, many specialists
Kling-MLLM is a shared autoregressive foundation model unifying image, video, audio, synchronized audio-video, and interleaved inputs. The family spans a dense 27B variant and MoE variants at 35B-A3B and 397B-A17B, with LLM and vision backbones initialized from Qwen3.5-Qwen3.8 checkpoints. Task-specific post-training specializes the base into captioners, prompt-enhancement models, text encoders, reward models, and evaluators. Audio towers are scale-dependent: smaller variants follow the Qwen3-Omni design, encoding 16-kHz monaural audio as 13 audio tokens per second; larger variants fuse Mel and waveform-VAE features in a dual-branch tower at 25 tokens per second. Training follows a four-stage curriculum: acoustic initialization, joint multimodal training, temporal-capacity expansion, and long-context consolidation.
Captions as a contract, rewards split in two
Because a DiT trains on far fewer samples than a language model, the report treats description density per unit of video duration as the core measure of data efficiency: captioning is a semantic contract, extended by reference-grounded captioning with explicit bindings. Prompt enhancement is layered. Basic PE translates user intent into the caption distribution, and a final DiT-in-the-loop RL stage connects PE to the frozen text encoder and DiT, updating it with video-level rewards for visual quality, motion quality, and text alignment. Agentic PE runs before basic PE at inference, handling story refinement and retrieval of compelling examples, because many production prompts are themselves AI-written: long, fluent, and internally inconsistent. Reward modeling splits in two. A General Reward Model built on Kling-MLLM-27B produces explainable, spatiotemporally localized critiques with defect tags, temporal intervals, per-frame boxes, and rationales; a Decision Reward Model scores many dimensions in a single prefill. Together they serve five production scenarios: DiT RL, PE RL, auxiliary machine evaluation, checkpoint selection, and data filtering.
So what
The text-encoder section deserves a close read: a reliable conditioning encoder comes from perceptual training rather than stronger language ability; general QA benchmarks measure autoregressive decoding, not the hidden states a DiT actually consumes; and candidate encoders can be screened from text alone before costly end-to-end training. Evaluation should follow the task, because a single aggregate score cannot diagnose a chain where the same failure may originate in captions that omit supervision, conditioning that discards temporal structure, a reward model that favors smooth but static motion, or an evaluator that never tests the requested event. The value of this report is not benchmark scores but turning understanding from a post-hoc judge into production infrastructure. arXiv report