Interactive world models have a failure mode that rarely makes headlines: the longer you walk through a generated world, the harder the machine has to work. Mainstream approaches either drag history frames or KV caches along as temporal context, or rebuild everything you generated into an explicit 3D memory such as point clouds or Gaussians. Both paths scale with the length of the generated stream. Past a certain distance you either run out of memory or sink below interactive frame rates. In the paper a KAIST-led team posted to arXiv on October 29, the temporal-context method WorldKV goes out-of-memory at 400 frames of generation, and Yume 1.5 needs 66.7 GiB to deliver 1.42 FPS.
Memory that scales with the scene, not the stream
The paper's answer is L3G, a Latent 3D Gaussian Memory: a memory updater (L3G-Mem) and a video diffusion transformer (L3G-Gen) alternate autoregressively; training uses DL3DV for L3G-Mem and DL3DV plus RealEstate10K for L3G-Gen, and the real-time L3G-World Fast comes from AnyFlow few-step distillation. Two design decisions carry the weight. First, memory is updated selectively: new Gaussians are spawned only in regions not yet covered, while existing ones receive residual refinement. Memory is not a buffer that only accumulates — it is a persistent scene representation that keeps being rewritten, so its size grows with the spatial extent of the scene (complexity O(S)) rather than with the number of generated frames (O(T·R)). Second, every Gaussian stores a VAE latent feature, so the memory renders directly into the video generator's latent space, skipping the render-to-RGB-then-re-encode round trip.
Rendering in latent space, two orders of magnitude faster
The second decision is where the speed comes from. Conditioning a latent video generator on an explicit 3D memory normally requires rendering RGB and re-encoding through the VAE, a round trip that rules out real time. The ablation quantifies the cost: remove direct latent storage, fall back to RGB rendering plus VAE encoding, and VAE-space rendering throughput collapses from 943.6 FPS to 2.97 FPS. Against the most efficient prior 3D-memory baseline for each metric, the paper reports 38.6x fewer 3D primitives, 3.2x faster memory updates, and 359x faster memory rendering.
One H100: 7.4 GiB, 9.02 FPS
The absolute numbers are just as telling. For 400-frame generation on a single H100, L3G-World peaks at 7.4 GiB and runs at 9.02 FPS overall — the only method in the comparison marked "real-time". NeoVerse, also Gaussian-based, needs 36.8 GiB for 1.99 FPS; the optimization-based ArtiFixer fits in 11.7 GiB but runs at 0.10 FPS. Across the DL3DV and RealEstate10K datasets and 8 baselines, the authors build 481-frame loop-closure trajectories — the same camera path walked forward, then retraced backwards — and compare frames at matching poses (PSNR/SSIM/LPIPS) to measure long-horizon self-consistency. L3G-World posts the best PSNR (16.00/18.36) and LPIPS, with 266 to 417x fewer 3D primitives than NeoVerse, and 8.9 to 9.5x faster reconstruction on 7.8 to 10.4x less memory than the best point-cloud baseline, Lyra 2.0.
Some cold water
Speed has a price. The few-step-distilled L3G-World Fast gives back some self-consistency, with PSNR falling from 16.00 to 14.59 on DL3DV. Generation resolution is 480×832, unremarkable for 2026. The system is assembled from existing components — Cosmos-Predict2.5, Depth Anything 3 — trained on 4 A100 GPUs, and the evaluated setting is a single input image plus a camera trajectory, still far from an open-world game.
For this track, though, the question the paper raises will outlive the paper itself: should the memory of a long-horizon generative system grow with time, or with space? The answer decides how far you can walk.
References: arXiv:2610.40614 · Project page