On July 27, 2026, arXiv posted a paper from Xuelong Li's team at Beihang — "MMOE: Modernizing Diffusion Transformers with Efficient Expert Design" (arXiv:2607.24665). It doesn't grab the spotlight like a model release, but it surfaces a question the AIGC community has quietly been avoiding: does a Diffusion Transformer really need to "pile up parameters" to get stronger?

The Core in One Sentence

MMOE is not as simple as "slapping a MoE layer onto DiT". What it does is systematically port the entire "efficiency toolbox" that LLMs used to scale over the past few years — routed experts, shared/lightweight experts, gate-residual routing, attention-residual information reuse — onto SiT-style diffusion transformers, and run ablations: when you add one more efficiency component, how does the convergence curve change, how does FID change, and what is the qualitative cost?

The experimental budget is deliberately modest: a single 8×H100 node, batch size 256, 400k steps. This is a setup any mid-sized lab can afford — not a resource play that only frontier labs can mount.

The Road LLMs Already Walked, That DiT Is Only Now Paving

Let's set the scene. LLMs scaled sustainably not because everyone kept "adding parameters", but because MoE decoupled "total parameters" from "activated parameters" — the total param count can be huge, but each token activates only a small slice, keeping compute in check. At the same time, design choices like mixed routed + shared experts, gate residuals, and attention residual information reuse — these "small switches" — collectively let LLM training converge stably without blowing up.

But on the DiT side, how has it gone? In the past two years, whether for image generation DiT or video DiT, once MoE was bolted on, the inertial move has been: first ramp up total parameters, then bump the sparsity ratio. Total params inflate fast, quality does go up, but single-step convergence curves among sparse variants keep beating each other, and the quality-cost trade-off has never been pinned down. In short, everyone has been playing the "parameter war", and nobody has carefully ported the LLM's already-validated efficiency mechanisms into AIGC and run a proper ablation.

MMOE exists to fill that gap.

What MMOE Actually Installs

The paper explicitly refuses to treat MoE as a "single plug-in". It splits the "modernized" expert design into four components and runs controlled experiments on the SiT backbone:

  1. Routed experts: the classical move — let each token's gating pick a top-k of experts. MMOE focuses on the stability of different routing strategies across DiT depth.
  2. Shared + Lightweight experts: alongside routing, add a "common pathway" for features that routing cannot stably categorize. Lightweight experts are cheaper shared variants that cut the param-inflation burden.
  3. Gate-residual routing: borrow LLM's residual-gate idea — add the gate's output to the trunk signal instead of replacing it. This way routing choices don't "overwrite" features the previous layer already polished, making convergence steadier.
  4. Attention-residual information reuse: inside the attention residual stream, reuse routing signals from earlier layers. This pulls up to the level of Transformer residual design itself, meaning expert scheduling across blocks can coordinate instead of operating in silos.

These four pieces aren't simply stacked — they are ablated in combination to see which one contributes most and which one is redundant.

Experimental Result: A Quality-Cost Win

The paper's core claim is unambiguous:

Under matched training + sampling protocol and identical budget, MMOE achieves lower FID at every checkpoint than both dense and intermediate sparse-expert baselines. In other words, it converges faster per training step.

Further, when comparing within the sparse variants (everybody using MoE), MMOE achieves the best quality-cost balance. That means — while the community keeps arguing "my model is bigger than yours", MMOE says: "my model is cheaper than yours, and the quality beats competitors with the same param count".

Routing analysis also reaches intuitive conclusions: expert specialization is stable across depth, lightweight routes see substantial use, and routing changes modestly across adjacent denoising steps. The design is not "MoE for the sake of MoE" — it really does assign different roles to different experts.

Why This Matters for the Industry

Zooming out, this paper matters on three axes.

First, mid-sized labs can now run it. An 8×H100, batch 256, 400k-step budget is roughly what any mid-sized team with a training budget can assemble. In the recent past, AIGC papers routinely announced "training consumed 1024 H100s for weeks"; most practitioners could only "read the paper", not "reproduce it". MMOE's budget settings compress the research-reproduction gap.

Second, it loosens the "parameter stacking" narrative. Today's AIGC community has an implicit consensus: model quality ≈ total parameters. If the MMOE line continues to validate, then "smarter methods that yield better quality-cost ratios" becomes a direction worth betting on — the AIGC counterpart of the LLM-MoE story.

Third, it's a tailwind for video DiT. Compute spend for video diffusion Transformers like Seedance, Veo, and Sora is dozens of times that of text-level diffusion. If MMOE-class designs can transfer to video DiT, inference costs drop materially, which advances both B-end usability and C-end real-time viability.

Personal Take: What This Paper's Posture Really Says

I'm reluctant to call this paper "disruptive". Putting MoE on diffusion transformers is not new, and MMOE isn't the first to do it. But it does something rare: it runs an ablation across every component of the LLM efficiency toolbox, with a reproducible setup and clear conclusions. In an AIGC environment increasingly dependent on "throw resources at it", that sense of restraint is itself a posture.

So what: if you run an image or video generation team and have been defaulting to "params = competitiveness", MMOE is your reminder that there is another path — systemically port LLM's proven efficiency designs onto your DiT, and ship models with better quality-cost ratios. If this direction gets validated at larger scale (more H100s, longer training), it could reshape the AIGC foundation-model investment logic over the next 12-18 months.

Source: arXiv:2607.24665, "MMOE: Modernizing Diffusion Transformers with Efficient Expert Design" submitted 27 Jul 2026.