arXiv 2606.19005 introduces Sumi, a 7B-parameter diffusion language model trained on 1.5T tokens. The standout: Sumi is the largest open diffusion LLM with a "native reference point" — i.e., it can be used as a starting point for downstream fine-tuning, evaluation, and deployment, just like an AR base model.
The "scaling diffusion LLMs" challenge: diffusion LLMs have historically been trained at small scales (1-3B parameters) and on limited data (100-500B tokens). The community lacked a "base model" comparable to Llama-3-7B or Qwen2.5-7B — a model that downstream developers can build on.
The Sumi fix: train a 7B diffusion LLM on 1.5T tokens (the same scale as Llama-3-7B), using a "masked diffusion" architecture with a novel "noise schedule" that improves training stability. The model is open-sourced with full training code and data pipeline.
The benchmark: Sumi-7B hits parity with Qwen2.5-7B on MMLU (within 1 point) and beats it on several code-generation tasks. The model is fully open-sourced, including the training data, training code, and evaluation scripts.
The bigger takeaway: "open diffusion LLM base model" is a critical missing piece. Sumi fills that gap, and the open-source release will accelerate the diffusion LLM research community. For the industry, this means diffusion LLMs are now a viable alternative to AR models for production deployments, with comparable quality and the additional benefits of diffusion (parallel generation, controllable generation, iterative refinement).