Interactive video world models have long been stuck on a contradiction: bidirectional video generators deliver high visual quality, but their multi-step denoising loops introduce latency that real-time interaction cannot tolerate. Causal distillation can compress sampling down to one or two steps, yet once you add game-native controls, discrete keyboard states and continuous mouse trajectories must stay frame-aligned with temporally compressed latent chunks during both causal training and autoregressive rollout — the moment alignment slips, control breaks. arXiv 2608.14022, a joint work by CUHK, Tencent PCG, Fudan University, Shanghai AI Laboratory and HKUST, proposes ForgeWM as a progressive answer, with code, weights, and data all released.
A Four-Stage Progressive Distillation Recipe
ForgeWM deliberately avoids a one-shot transformation. Instead, it turns a bidirectional action-conditioned generator into causal students across four stages. Stage 0 performs bidirectional domain-adaptation fine-tuning on the target game, then freezes the model as the teacher for later stages. Stage 1 applies teacher-forced block-wise causal attention to make the generator autoregressive without sacrificing control fidelity. Stage 2 uses causal consistency distillation to compress the sampling trajectory and unlock few-step generation. Stage 3 lets the student roll out on its own while a frozen bidirectional teacher supervises at the distribution level. The output is a family of budget-specialized students running at 1, 2, and 4 denoising steps. The base model is the open-source Matrix-Game 2, with training data from GameFactory Minecraft clips and SCOPE FPS data.
One Denoising Step, 72 FPS
On an evaluation of 1,000 paired Minecraft trajectories, the paper reports ForgeWM-1 reaching 72.10 FPS with 168.2 ms per-chunk latency (352×640, measured on a single H20, excluding loading and VAE decoding), versus 370.9 ms and 32.35 FPS for the distilled Matrix-Game 2 baseline. On the quality side, the authors report leading scores on six of seven quality and control metrics: ForgeWM-2 tops the table with 0.6865 Imaging Quality, 0.9740 keyboard-control accuracy, and 0.8268 mouse-control accuracy. In a blind study, 41 participants made 615 three-way blind selections, with ForgeWM-4 taking 68.8% of visual-quality preferences and 60.7% pooled.
Replay-Time Refinement: One-Step Draft, Four-Step Quality
The system's cleverest design is its dual-path deployment: run the one-step student during interaction to protect latency; after the session ends, that same student re-noises its saved draft latents and refines them chunk by chunk under the recorded action sequence. The refined replay scores 0.6155 LPIPS against 0.6168 for direct four-step generation — essentially matching quality — while staying roughly three times closer to the trajectory actually played (0.1970 versus 0.6187). The refiner is the deployed model itself: no extra weights, no online compute cost.
Reproducible on 8 GPUs, and the Recipe Transfers
All four Minecraft-stage checkpoints, the few-step students, and the CrossFPS inference model, plus 40,000 pre-encoded GF-Minecraft training clips, are on HuggingFace, and the paper states the full recipe reproduces on 8 GPUs. The same four-stage pipeline transfers to seven FPS games by swapping in gamepad buttons, dual analog sticks, and SCOPE data — no architectural change — reaching a macro-average paired LPIPS of 0.656. For anyone building interactive generation, the takeaway is direct: the reproduction barrier for real-time world models has dropped to academic-lab scale — 8 GPUs, open data, and runnable scripts are all in place. The next battleground is not raw compute, but the fine craft of control alignment and distillation strategy. Paper