JD.com has open-sourced JoyAI-Video-Edit on GitHub — a real-time, instruction-guided video editing system aimed at open-ended video streams. The model combines an MLLM-based condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion Transformer (MMDiT). Through autoregressive diffusion plus distillation, it turns "edit videos with natural language as they play" from an offline batch task into a real-time streaming generation workflow.

Key capabilities

  • Real-time open-ended editing. For live camera streams or uploaded videos, the model edits frames causally as they arrive — no need to know the total length up front, and no need to revisit future frames.
  • Diverse instruction control. Supports subject edits, local edits, background replacement, style transfer, motion changes, and reference-guided editing.
  • High-throughput 720p deployment. The official deployment benchmark shows the end-to-end pipeline reaching 30.19 FPS at 720×1280, pushing video editing from offline batch processing into interactive streaming generation.
  • Train-inference consistency. Aligned autoregressive distribution matching distillation, long-horizon optimization, and bounded KV-state inference mitigate train-inference mismatch and accumulated temporal drift.

Architecture

The three core components divide the work cleanly:

  • MLLM condition encoder encodes the user's natural language instructions (and reference images) into condition signals for the downstream diffusion Transformer.
  • Causal video VAE performs spatiotemporal compression and reconstruction in the token space — this is what makes "real-time streaming" viable, because the model can't afford to recompute the whole video every frame.
  • 16B MMDiT backbone performs diffusion-style editing on the compressed token sequence.

On the deployment side, the project ships with persistent TorchInductor / Triton / CUDA cache reuse, bounded KV-state scheduling, and a stable per-chunk compute budget — the goal is to keep 720p editing consistent under load.

Release and availability

JD.com released the full deployment code, technical report (arXiv:2608.03974), online demo (joyai-labs.jd.com/v2v), and Hugging Face weights under the Apache 2.0 license. The repository's DEPLOYMENT.md covers environment setup, checkpoint preparation, and custom deployment. The default server listens on http://localhost:8080.

The GitHub TODO also points at the next steps:

  • Consumer-GPU deployment optimization (e.g. GeForce RTX 5090).
  • A stronger version is in training, with a focus on reference-image-guided video editing (RV2V).
  • Full training and data pipeline open-sourcing.

Why it matters

The "last mile" of video generation is shifting from "offline editing" to "live editing." JoyAI-Video-Edit turns that shift into a deployable form through the combination of autoregressive diffusion and a streaming VAE — 30.19 FPS end-to-end at 720p is a number that holds up in engineering terms, and crucially the team released weights, a training report, and deployment code, not just a paper.

Two signals worth watching:

  • A big-tech bet on a physical-world model matrix. JD.com's framing — "the physical-world model matrix continues to grow" — and the explicit link to embodied-AI data synthesis suggest that real-time editing + causal streaming is a key building block of the next-generation synthetic data pipeline.
  • The "real-time" inflection in multimodal diffusion. The Sora / Veo generation was about "quality." Now that a 16B model can run at 30+ FPS at 720p, real-timeness is the next differentiator. For individual creators and live e-commerce, that means "change the shot" stops being a post-production step and becomes something you do live.

What to watch next: consumer-GPU support and stronger RV2V reference-image capability. Whoever nails those two first gets to move real-time video editing from demo to daily tool.

References