arXiv 2606.19475 introduces the first "unified benchmark" for diffusion language models (DLMs), evaluating 8 leading DLMs across 8 tasks. The standout finding: the DLM space is highly fragmented — there's no "one DLM rules all," and each model has a specific niche.
The benchmark structure: 8 tasks spanning language modeling, code generation, math reasoning, long-form Q&A, multilingual translation, instruction following, creative writing, and chain-of-thought reasoning. Each task is evaluated with 500 standardized prompts, and the score is averaged across the 8 tasks. The 8 models evaluated: LLaDA, LLaDA-2, SEDD, MDLM, iLLaDA, Plaid, BD3-LM, and DiffuGPT.
The result: the leaderboard is highly fragmented. SEDD leads on language modeling and creative writing; LLaDA-2 leads on code generation; MDLM leads on long-form Q&A; iLLaDA leads on chain-of-thought reasoning. There's no model that wins on all 8 tasks. The "average rank" varies from 1.5 (best) to 7.2 (worst), with most models in the 3-5 range.
The "niche specialization" finding: the authors analyze why each model wins on its niche. SEDD's strength is its "absorbing diffusion" formulation, which is better for low-entropy tasks (language modeling, creative writing). LLaDA-2's strength is its "masked diffusion" + SFT pipeline, which is better for code (where structure matters). MDLM's strength is its "continuous-time" formulation, which is better for long-form generation (where temporal coherence matters).
The bigger takeaway: "DLM benchmark fragmentation" is a wake-up call for the DLM community. The "one model wins all" assumption (which dominated the AR era) doesn't hold for DLMs, and the field needs to develop a more nuanced understanding of which DLM is best for which task. For the industry, this means DLM deployment will likely involve a "model router" that picks the right DLM per task.