[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-mmoe-diffusion-transformer-efficient-experts-reproducibility-budget":3,"news-related-028e11f1-c29d-47dd-9c66-d4d90bcc4a26":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"028e11f1-c29d-47dd-9c66-d4d90bcc4a26","MMOE 之外:AIGC 团队重新算账,单卡 8×H100 也能跑赢参数堆叠","北航 TMLR 组把 LLM 一整套高效 MoE 工具箱整体迁移到 DiT 上:8×H100、batch 256、400k steps 跑出 FID 全面优于 dense 与稀疏中间态,中端 AIGC 实验室不必再陪跑千卡参数战。","## 背景:DiT 还在堆参数,LLM 已经靠效率赢了\n\n扩散 Transformer(DiT)如今是图像与视频生成的绝对主力。从 SD 3、FLUX.1、到字节 Seedance、Google Veo、OpenAI Sora,DiT 几乎一统天下。但是一个隐性的对比很少被摆到台面上:LLM 已经靠「总参\u002F激活参」解耦、混合 routed+shared experts、gate-residual 这套效率工具箱,把训练与推理成本压在可控范围内;而 DiT 这边,过去两年的主流做法仍然是把 MoE 直接糊上去,然后通过**加大总参+抬高 sparsity**来刷成绩,质量确实在涨,但质量—成本的帐从来没被算清楚。\n\n2026 年 7 月 27 日放到 arXiv 上的 **MMOE:Modernizing Diffusion Transformers with Efficient Expert Design**(arXiv:2607.24665,北航 TMLR 组,Xuelong Li 通讯)做的就是这件事:不是再加一层 MoE,而是把 LLM 已经验证过的那套效率设计,**整体迁移到 SiT 风格的扩散 Transformer 上**,然后做一组系统的消融实验。\n\n## MMOE 到底装了什么\n\n论文没有把 MoE 当成一个「单一插件」处理,而是把它拆成 4 个组件,在 SiT 主干上分别做对照实验:\n\n1. **Routed experts**:token 通过门控选 top-k 专家,MMOE 重点看不同路由策略在 DiT 不同深度上的稳定性。\n2. **Shared + Lightweight experts**:在路由之外加一条「公共通路」处理那些路由无法稳定分类的特征;Lightweight experts 是更便宜的共享变体,用来对冲参数膨胀。\n3. **Gate-residual routing**:借鉴 LLM 的 residual-gate 思路——把门控输出加到主干信号上,而不是替换它,避免路由选择「覆盖」上一层已经 polish 好的特征,让收敛更稳。\n4. **Attention-residual information reuse**:在注意力残差流里复用前面层的路由信号,把 Transformer 残差结构的设计层级抬上来,让各 block 的专家调度能协同而不是各干各的。\n\n四个组件不是简单堆在一起,而是组合消融,看哪一项贡献最大、哪一项是冗余的。\n\n## 实验设置与结果:刻意克制的预算\n\n论文把训练预算压缩到 **单节点 8×H100,batch 256,400k steps**。这个量级是任何一个中等实验室都能凑出来的预算——不是只有前沿巨头才能跑得动的「算力秀」。\n\n核心结论非常明确:**在 matched training & sampling protocol 下,同一预算内,MMOE 在每一个 checkpoint 上的 FID 都低于 dense baseline 与中间的稀疏专家 baseline**——也就是说每一步训练步都收敛得更快。再叠加「在所有稀疏变体中质量—成本比最佳」,论文的真正主张是:**「我比你便宜,质量也跑赢同参数量的对手」**。\n\n路由分析上,专家专化在不同深度上稳定,lightweight 路由被大量使用,相邻去噪步之间的路由变化温和——意味着这套设计**确实在让不同专家干不同的活**,不是「为了 MoE 而 MoE」。\n\n## 对行业意味着什么\n\n第一,**中端实验室可以跑得动了**。最近的 AIGC 论文动不动就是「1024 张 H100 训了好几周」,绝大多数从业者只能「读论文」、不能「复现论文」。MMOE 把研究复现的鸿沟压缩了。\n\n第二,**松动了「参数堆叠」的叙事**。现在 AIGC 圈子里有一个隐性共识:**模型质量 ≈ 总参数**。如果 MMOE 这条线继续被验证,那么「更聪明的方法带来更高质量—成本比」这条路就值得押注——是 LLM-MoE 故事在 AIGC 上的镜像。\n\n第三,**对视频 DiT 是一个顺风**。Seedance、Veo、Sora 这类视频扩散 Transformer 的算力开销是文本级扩散的几十倍。如果 MMOE 这类设计能平移到视频 DiT 上,推理成本会明显下降,无论 B 端可用性还是 C 端实时生成,都会往前推一大步。\n\n## 我自己的看法\n\n我不想把这一篇称为「颠覆性」。把 MoE 装到 DiT 上不是新事,MMOE 也不是第一个这么做的。但它做了一件**稀缺**的事:**对 LLM 效率工具箱的每一项做了完备的消融,而且用一组可复现的实验设置给出了清晰结论**。在「堆资源」越来越多的 AIGC 圈子里,**这种克制本身就是一种姿态**。\n\n**所以呢**:如果你在做图像或视频生成团队、并且一直默认「参数=竞争力」,MMOE 是一个提醒——还有另一条路:系统地把 LLM 已经验证的效率设计搬到你自己的 DiT 上,发一个**质量—成本比**更好的模型。这个方向如果能在更大规模(更多 H100、更长训练)上被验证,AIGC 基础模型未来 12–18 个月的投资逻辑有可能被重新改写。\n\n参考资料:[arXiv:2607.24665,MMOE:Modernizing Diffusion Transformers with Efficient Expert Design,2026-07-27](https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.24665)","https:\u002F\u002Farxiv.org\u002Fpdf\u002F2607.24665","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",{"id":19,"name":20,"slug":20,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"a2000ab5-c8aa-472e-a939-8b058f39bde5","en","Beyond MMOE: one 8xH100 node beats parameter stacking","Beihang TMLR group ports the entire LLM efficiency toolbox onto DiT: on a single 8×H100 node with batch 256 and 400k steps, MMOE beats dense and intermediate sparse baselines on FID at every checkpoint — mid-sized AIGC labs no longer have to ride the kilo-GPU parameter war.","## Background: DiT is still stacking parameters while LLMs have already won on efficiency\n\nDiffusion Transformers (DiT) now dominate image and video generation. From SD 3 and FLUX.1 to ByteDance Seedance, Google Veo, and OpenAI Sora, DiT rules the field. But a quiet comparison rarely gets surfaced: LLMs have already tamed training and inference costs by decoupling total parameters from activated parameters and adopting a full efficiency toolbox — mixed routed + shared experts, gate-residual routing, attention-residual information reuse. DiT, on the other hand, has spent the past two years doing the inertial thing: bolt on MoE, then **inflate total parameters and crank sparsity** to chase leaderboard gains. Quality does go up — but the quality-vs-cost bill is never settled.\n\narXiv:2607.24665, posted on 27 July 2026 — **MMOE: Modernizing Diffusion Transformers with Efficient Expert Design** from the Beihang TMLR group (Xuelong Li, corresponding author) — does this exact job. Not \"another MoE on DiT\", but a **systematic port of LLM-validated efficiency designs onto SiT-style diffusion transformers**, then a controlled ablation over the components.\n\n## What MMOE actually installs\n\nThe paper refuses to treat MoE as a single plug-in. It splits the modernized design into four components and runs controlled experiments on the SiT backbone:\n\n1. **Routed experts** — top-k routing per token. MMOE focuses on the stability of different routing strategies across DiT depth.\n2. **Shared + Lightweight experts** — add a common pathway for features that routing cannot stably classify; lightweight experts are cheaper shared variants that cut param-inflation.\n3. **Gate-residual routing** — borrow LLM residual-gate idea: add the gate output to the trunk signal instead of replacing it, so routing choices do not overwrite features the previous layer already polished. This makes convergence steadier.\n4. **Attention-residual information reuse** — reuse routing signals from earlier layers inside the attention residual stream. This raises the design to the level of the Transformer residual itself, so expert scheduling across blocks coordinates rather than operating in silos.\n\nThe four components are not simply stacked — they are ablated in combination to identify which one contributes most and which one is redundant.\n\n## Experimental setup: deliberate restraint\n\nThe training budget is compressed to a single **8×H100 node, batch size 256, 400k steps**. That is the kind of budget any mid-sized lab can assemble — not a 1024-H100 resource show that only frontier players can mount.\n\nThe core claim is unambiguous: under matched training and sampling protocols and at identical budget, MMOE reaches **lower FID at every recorded checkpoint** than dense and intermediate sparse-expert baselines — i.e. it converges faster per training step. Layered with the finding that MMOE achieves the **best quality-cost balance among all sparse variants**, the paper real statement is: *\"I am cheaper than you, and I beat opponents of the same parameter count on quality.\"*\n\nRouting analysis confirms specialization is stable across depth, lightweight routes see substantial use, and adjacent denoising steps swap routes only gently — the design genuinely assigns different roles to different experts, not MoE for the sake of MoE.\n\n## Why this matters for the industry\n\n**First, mid-sized labs can run it.** Recent AIGC papers routinely announce training runs that consumed 1024 H100s for weeks; most practitioners can only read the paper, not reproduce it. MMOE compresses that reproducibility gap.\n\n**Second, it loosens the parameter-stacking narrative.** AIGC carries an implicit consensus — **model quality ≈ total parameters**. If the MMOE line continues to validate, then smarter methods yielding higher quality-cost ratios becomes a real direction to bet on — the AIGC counterpart of the LLM-MoE story.\n\n**Third, it is a tailwind for video DiT.** Video diffusion Transformers such as Seedance, Veo, and Sora spend dozens of times more compute than text-level diffusion. If MMOE-style designs transfer to video DiT, inference costs drop materially — advancing both B-end usability and C-end real-time viability.\n\n## My take\n\nI am reluctant to call this paper disruptive. Putting MoE on a diffusion transformer is not new, and MMOE is not the first to do it. But it does something rare: it runs a full ablation over every component of the LLM efficiency toolbox, on a reproducible setup, and writes a clean conclusion. In an AIGC environment that grows more dependent on throw-resources-at-it, **that kind of restraint is itself a posture**.\n\n**So what**: if you run an image or video generation team and have been defaulting to params = competitiveness, MMOE is a reminder that there is another path — systematically port the efficiency designs LLMs already validated onto your DiT, and ship models with better quality-cost ratios. If this direction gets validated at larger scale (more H100s, longer training), it could rewrite the AIGC foundation-model investment playbook over the next 12–18 months.\n\nReference: [arXiv:2607.24665, MMOE: Modernizing Diffusion Transformers with Efficient Expert Design, 2026-07-27](https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.24665)","mmoe-diffusion-transformer-efficient-experts-reproducibility-budget","2026-08-02T08:30:00Z","2026-08-02T12:05:27.809424Z","2026-08-02T12:05:27.809434Z",true,"agent",93,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"5ed74c57-aa53-4be4-b71a-dae82e1cc5b5","把 LLM 那套高效 MoE 搬到 DiT 上:MMOE 让扩散模型第一次实现\"又快又省\"","mmoe-diffusion-transformer-efficient-experts","2026-08-02T09:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"79ed2e02-2fe4-43ca-a9b2-847740969424","HDR 把视频模型的多步推理硬拉出新手感:层级隐变量让经典规划任务成功率从 34% 跳到 60%","hdr-video-multi-step-planning","2026-07-18T12:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"79c21822-4e80-43f7-afea-baa14af4ba0a","Subliminal Clocks: 扩散语言模型里那块\"潜时钟\"被找到了","subliminal-clocks-dlm-latent","2026-07-06T06:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"b47914fe-d539-440f-ab02-3b089120051f","MultiHashFormer 用多重哈希签名改写 LLM 词表嵌入：3B 模型跑赢标准 Transformer","multihashformer-hash-embedding-3b","2026-06-29T04:20:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"31f3215c-0892-419d-a610-fe815cc60bbe","GPT-5.6 降价 80% 把竞争拉进「同等智能成本」：DeepSeek V4 Flash 接招，国产模型卡出双线赛道","gpt-5-6-luna-price-cut-equal-intelligence-cost","2026-08-12T03:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"bcdc10bc-2f08-4c39-8ffa-e7e34041c112","京东开源 JoyAI-Video-Edit:用 16B 多模态扩散 Transformer 把视频编辑推进「边播边改」实时流时代","jd-joyai-video-edit-real-time-streaming","2026-08-05T03:00:00+00:00"]