arXiv 2606.24888 introduces DiffusionBench, a comprehensive benchmark that evaluates 21 diffusion Transformer models on a standardized set of 12 image and video generation tasks. The most striking finding: the ImageNet FID leaderboard is a poor predictor of real-world generation quality — the top-3 ImageNet models rank 7th, 11th, and 14th on the DiffusionBench composite score.
The benchmark structure: 12 tasks spanning image generation, image editing, video generation, video editing, controllable generation, and conditional generation. Each task has a standardized evaluation pipeline (same prompts, same seeds, same evaluation metrics), eliminating the variability of "in-house" benchmarks.
The findings: (1) ImageNet FID is saturated — the top-20 ImageNet models all have FID within 1 point, but their real-world quality varies dramatically. (2) Some "niche" models (e.g., a model trained on COCO only) outperform "generalist" models on specific tasks. (3) The best diffusion Transformer for video is not the same as the best for image — there is no "one model rules all." (4) Training data quality matters more than model size — a 1B model on high-quality data can beat a 5B model on web-scraped data.
The bigger takeaway: the "leaderboard-driven" evaluation paradigm in diffusion models is broken. DiffusionBench is a wake-up call — the industry needs standardized, multi-task benchmarks, not single-task leaderboards. For the industry, this means the next round of model releases should be evaluated on DiffusionBench-style comprehensive benchmarks, not just ImageNet FID.