Few-step autoregressive video diffusion is the mainstream route to streaming generation: a video is factorized into ordered temporal blocks, diffusion runs inside each block, and completed blocks are cached and reused to condition future generation. That causal, streaming interface is what interactive applications want. But pushing it from "works" to "real-time" has mostly meant post-training distillation, and the cost of that route has stayed stubbornly high.
Why three-network distillation is expensive
DMD-based methods such as Self Forcing and Causal Forcing maintain three video diffusion networks during each training update. The paper lays out the bill: distilling a Wan 2.1 1.3B generator typically requires an additional frozen 14B teacher and an online 1.3B critic. The teacher supplies the target-distribution score, the critic fits the student-distribution score, and only their difference yields a training signal. Maintaining and evaluating those networks makes every update expensive in both compute and memory.
The ViRDM idea: match a precomputed distribution
ViRDM, from Northeastern University and Adobe Research (first author Zichong Meng, done during an internship at Adobe Research), flips the question: can we drop the teacher and the critic entirely, and post-train only the generator against a precomputed target distribution? The idea transfers representation distribution matching (RDM) from one-step image generation, but a literal transfer hits three walls: a memory-intractable end-to-end gradient path, an image-derived optimization recipe that fails on video, and representation distributions that underconstrain temporal dynamics.
The recipe dismantles the walls one by one. Stochastically truncated clean exits select one denoising step per update, so only that step's clean endpoint prediction is backpropagated. A lightweight VAE decoder and staged vector-Jacobian products keep the generator, decoder, and representation encoder backward graphs from coexisting in memory. The team also maps out a video-specific regime: fresh batches of 8 to 64 generated videos already suffice, versus the broad optimum above 2,048 samples reported for image RDM. Initialization must come from a causally adapted model, representation distribution matching can refine an existing causal generator but cannot convert a bidirectional checkpoint into a causal one. A lightweight dynamics regularization closes the remaining gap on motion.
The numbers: 20 updates, 16 A100 GPU-hours
The paper reports 84.87 Total on the official VBench evaluation, 0.36 above the previous best few-step causal baseline, Causal Forcing, with Dynamic Degree at 72.02. Without the Dynamic Degree dimension, the model reaches 85.77 Total, surpassing the 85.34 previously achieved by Self Forcing. On the cost side: two hours on eight A100 GPUs (16 GPU-hours), 20 generator updates at roughly six minutes each, peak memory of 48.3 GB per GPU, and a single 80 GB A100 configuration with gradient accumulation peaking at 68.5 GB. The same recipe extends further: with one denoising step for subsequent blocks it obtains the best Total of 84.27 in that setting, and on bidirectional generation with one, two, and four steps it scores 83.12/84.53/84.56.
Where to stay skeptical
The paper's own limitations section is blunt: this is a post-training recipe, not a replacement for causal pretraining; primary experiments cover only the Wan2.1-1.3B backbone and 81-frame videos, leaving larger scales, higher resolutions, and longer horizons untested; and the method inherits the inductive biases of its frozen representation encoder and fixed reference population. The "+0.36 over the baseline" figure is the authors' own comparison under their experimental setup, so third-party replication is still pending.
For video-generation engineers, the real signal is this: distillation is no longer the only game that requires stacking three networks on a cluster. Representation distribution matching pushes the entry ticket for few-step causal post-training down to single-GPU territory. The next thing to watch is replication on larger backbones — if the 16-GPU-hour order of magnitude holds, the cost curve of few-step causal video generation gets pushed down another notch.
Reference: arXiv:2609.28923 (https://arxiv.org/abs/2609.28923), project page neu-vi.github.io/ViRDM/