Background: DiT is still stacking parameters while LLMs have already won on efficiency

Diffusion Transformers (DiT) now dominate image and video generation. From SD 3 and FLUX.1 to ByteDance Seedance, Google Veo, and OpenAI Sora, DiT rules the field. But a quiet comparison rarely gets surfaced: LLMs have already tamed training and inference costs by decoupling total parameters from activated parameters and adopting a full efficiency toolbox — mixed routed + shared experts, gate-residual routing, attention-residual information reuse. DiT, on the other hand, has spent the past two years doing the inertial thing: bolt on MoE, then inflate total parameters and crank sparsity to chase leaderboard gains. Quality does go up — but the quality-vs-cost bill is never settled.

arXiv:2607.24665, posted on 27 July 2026 — MMOE: Modernizing Diffusion Transformers with Efficient Expert Design from the Beihang TMLR group (Xuelong Li, corresponding author) — does this exact job. Not "another MoE on DiT", but a systematic port of LLM-validated efficiency designs onto SiT-style diffusion transformers, then a controlled ablation over the components.

What MMOE actually installs

The paper refuses to treat MoE as a single plug-in. It splits the modernized design into four components and runs controlled experiments on the SiT backbone:

  1. Routed experts — top-k routing per token. MMOE focuses on the stability of different routing strategies across DiT depth.
  2. Shared + Lightweight experts — add a common pathway for features that routing cannot stably classify; lightweight experts are cheaper shared variants that cut param-inflation.
  3. Gate-residual routing — borrow LLM residual-gate idea: add the gate output to the trunk signal instead of replacing it, so routing choices do not overwrite features the previous layer already polished. This makes convergence steadier.
  4. Attention-residual information reuse — reuse routing signals from earlier layers inside the attention residual stream. This raises the design to the level of the Transformer residual itself, so expert scheduling across blocks coordinates rather than operating in silos.

The four components are not simply stacked — they are ablated in combination to identify which one contributes most and which one is redundant.

Experimental setup: deliberate restraint

The training budget is compressed to a single 8×H100 node, batch size 256, 400k steps. That is the kind of budget any mid-sized lab can assemble — not a 1024-H100 resource show that only frontier players can mount.

The core claim is unambiguous: under matched training and sampling protocols and at identical budget, MMOE reaches lower FID at every recorded checkpoint than dense and intermediate sparse-expert baselines — i.e. it converges faster per training step. Layered with the finding that MMOE achieves the best quality-cost balance among all sparse variants, the paper real statement is: "I am cheaper than you, and I beat opponents of the same parameter count on quality."

Routing analysis confirms specialization is stable across depth, lightweight routes see substantial use, and adjacent denoising steps swap routes only gently — the design genuinely assigns different roles to different experts, not MoE for the sake of MoE.

Why this matters for the industry

First, mid-sized labs can run it. Recent AIGC papers routinely announce training runs that consumed 1024 H100s for weeks; most practitioners can only read the paper, not reproduce it. MMOE compresses that reproducibility gap.

Second, it loosens the parameter-stacking narrative. AIGC carries an implicit consensus — model quality ≈ total parameters. If the MMOE line continues to validate, then smarter methods yielding higher quality-cost ratios becomes a real direction to bet on — the AIGC counterpart of the LLM-MoE story.

Third, it is a tailwind for video DiT. Video diffusion Transformers such as Seedance, Veo, and Sora spend dozens of times more compute than text-level diffusion. If MMOE-style designs transfer to video DiT, inference costs drop materially — advancing both B-end usability and C-end real-time viability.

My take

I am reluctant to call this paper disruptive. Putting MoE on a diffusion transformer is not new, and MMOE is not the first to do it. But it does something rare: it runs a full ablation over every component of the LLM efficiency toolbox, on a reproducible setup, and writes a clean conclusion. In an AIGC environment that grows more dependent on throw-resources-at-it, that kind of restraint is itself a posture.

So what: if you run an image or video generation team and have been defaulting to params = competitiveness, MMOE is a reminder that there is another path — systematically port the efficiency designs LLMs already validated onto your DiT, and ship models with better quality-cost ratios. If this direction gets validated at larger scale (more H100s, longer training), it could rewrite the AIGC foundation-model investment playbook over the next 12–18 months.

Reference: arXiv:2607.24665, MMOE: Modernizing Diffusion Transformers with Efficient Expert Design, 2026-07-27