Background

Unified multimodal models (UMMs) have long run two parallel visual interfaces — a ViT for understanding-side semantic features and a VAE for generation-side reconstruction latents. Visual context length is doubled, and every new task has to be wired into both pipelines. Pixel-space modeling has been tried on images, but extending that paradigm to a single interface for both image and video understanding and generation has not been pulled off by anyone — until now.

NVIDIA's PixelUMM (arXiv:2609.38597, GitHub nv-tlabs/PixelUMM, weights mirrored at Hugging Face nvidia/PixelUMM), posted September 29, 2026, takes that on. Authors: Cong Wei, Xuanchi Ren, Bryan Chu, Weiming Ren, Huan Ling, Jiahui Huang, Laura Leal-Taixé, Sanja Fidler, Wenhu Chen, Zian Wang, Jay Zhangjie Wu.

Core idea

PixelUMM has no VAE and no vision encoder — a single decoder-only Transformer reads and writes raw pixels. Images are split into 16×16 patches, videos into 4-frame spatio-temporal tubelets, and both go through single-layer linear projections into a shared multimodal backbone (Qwen3-8B). A Mixture-of-Transformers split separates task-specific parameters, with one set of self-attention shared across text, clean pixels, and noisy pixels. The model jointly supports autoregressive text prediction and pixel-space flow matching — no separate VAE-token stream, no separate understanding branch.

The crucial detail is that clean pixels, noisy pixels, and text all share one self-attention window. The paper's F7 experiment: after 15K steps of multi-task fine-tuning, image understanding moves both ways (BLINK +2.37, CV-Bench +2.27, SEED-I −5.41), while all four video benchmarks improve by 0.89–3.49 points. That tells you clean pixel conditions routing through the understanding expert do not systematically hurt understanding — the most obvious concern once you drop the VAE.

Results

PixelUMM reports 1.7B and 8B models trained with the same recipe (global batch 256). Headlines:

  • 8B reaches MSE 0.060 at ~14K steps; 1.7B needs ~40K. Text CE 0.50 lands at ~10K for 8B vs ~31K for 1.7B — 8B reaches comparable generation and text losses in roughly one third of the training steps.
  • 128 GPUs vs 8 GPUs on the same 1.7B model: text CE reaches 1.0 at ~2K steps (vs ~17.5K at 8 GPUs), a ~9× speedup; generation MSE reaches 0.065 at ~17K (vs ~23K), a ~1.3× speedup. More GPUs accelerate text far more than generation.
  • Video tubelet compression: p32/t1 (1024 pixels/token) gives the lowest T2V loss; p16/t4 (4096 pixels/token) is used in the final model to align with common video VAEs like Wan2.2; p32/t4 (8192 pixels/token) raises it back up — "stronger compression makes video generation harder to learn."
  • Patch size: 32×32 packs 4× more images per step but T2I loss stays above 16×16 in late training; final model uses 16×16.
  • Convolutional output heads: Wan-style upsample+conv has the cleanest boundaries but 17.1× GFLOPs; PixelShuffle temporal-first (F3-R03) converges T2V pixel-flow loss to ~0.02 with smaller boundary residual elevation. The released checkpoint and online demos still use linear heads — switching requires additional training. Understanding-benchmark tables put 8B PixelUMM on par with BAGEL-7B, TUNA-7B+5B, LLaVA-OV-1.5/28B, Qwen3-VL-Inst-8B across BLINK, MMStar, SEED-I, DocVQA, CountBench, MMVP, etc. — wins and losses mixed (the paper explicitly notes that different training data preclude declaring an architecture winner). On video the same model in sparse_mode samples 96 frames and posts Video-MME (no subtitles) 57.33, behind LLaVA-OV-28B (71.90) and Qwen3-VL-8B (71.40), but sparse_mode/dense_mode at 4 FPS or 1 FPS are within ±0.7 on MVBench/Video-MME/LongVideoBench/LVBench.

Why it matters

Dropping the VAE is not free. The paper itself calls out the limitations: many-entity scenes where overlapping bodies drift in and out, hands with wrong finger counts, and physically implausible motion — PixelSPACE mode does not bypass the failure modes latent video diffusion models have. The price of keeping the VAE is also real: doubled context length, dual training pipelines, alignment across two visual interfaces. The paper's experiments show 8B reaches comparable loss in roughly one third the steps of 1.7B, and 128 GPUs speed up text by ~9× while speeding generation by only ~1.3× — "text converges fast, pixels converge slowly" is the scaling observation most worth watching on this architecture. If the convolutional head is properly trained through (the paper already points at PixelShuffle temporal-first as the better trade-off), video side should pick up another segment of gain.

For practitioners, PixelUMM collapses two visual interfaces (ViT tokens + VAE tokens) into one pixel token stream, which is worth tracking for teams that already build on Qwen3 multimodal pipelines and do not want to maintain two visual interfaces in production. Training stability of pixel-space flow matching and boundary artifacts in long video sequences are the production-blocking items to measure.

Sources: