[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-mmoe-diffusion-transformer-efficient-experts":3,"news-related-5ed74c57-aa53-4be4-b71a-dae82e1cc5b5":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"5ed74c57-aa53-4be4-b71a-dae82e1cc5b5","把 LLM 那套高效 MoE 搬到 DiT 上:MMOE 让扩散模型第一次实现\"又快又省\"","arXiv:2607.24665 提出 MMOE,把 routed experts、shared\u002Flightweight experts、gate-residual routing、attention-residual 信息复用系统性搬到 SiT 风格扩散 Transformer 上,在单台 8×H100 上让每一步的 FID 都低于 dense 基线,在稀疏变体里质量-成本比最好,展示了 AIGC 基础模型走 LLM 那条平衡扩展路径的可行性。","2026 年 7 月 27 日,arXiv 上线了一篇来自北航 Xuelong Li 团队的新论文 ——《MMOE: Modernizing Diffusion Transformers with Efficient Expert Design》(arXiv:2607.24665)。它不抢模型发布的聚光灯,却把一个被整个 AIGC 圈子默默回避的问题摆到了台面上:**扩散 Transformer 真的需要靠\"堆参数\"来变强吗?**\n\n## 一句话核心\n\nMMOE 不是\"给 DiT 加一个 MoE 层\"那么简单。它做的事情是把过去几年 LLM 走向规模化的整套\"效率工具箱\"——routed experts、shared\u002Flightweight experts、gate-residual routing、attention-residual information reuse——**系统性地**移植到 SiT 风格扩散 Transformer 上,然后做了 ablation:每多塞进去一块效率组件,收敛曲线怎么变,FID 怎么变,qualitative cost 又怎么变。\n\n实验预算控制得非常克制:**单台 8× H100, batch size 256, 跑满 400k 步**。这是任何一家中型实验室都能复现的资源,不是只有头部玩家玩得起。\n\n## LLM 已经走过的路,DiT 现在才补上\n\n先看背景。LLM 这一波能持续扩展,不是因为大家都在\"加参数\",而是因为 MoE 路线把\"激活参数\"和\"总参数\"解耦了——总参数可以很大,但每个 token 只激活其中一小部分,算力开销被压下来。同时 routed + shared 的混合专家设计、gate 的残差连接、attention 残差的信息复用这些\"小开关\",共同让 LLM 训练能够稳定地收敛而不爆炸。\n\n但 DiT 这边怎么做的?近两年,无论是图像生成的 DiT、还是视频生成的视频 DiT,一旦塞进 MoE,大家的惯性动作是:**先把总参数拉满,再拉 sparsity 比例**。参数膨胀得很快,质量也确实在涨,但 single-step 收敛曲线稀疏变体之间互有输赢,质量-成本的折中关系一直说不清楚。简单说,大家都在做\"参数战\",没人认认真真把 LLM 那套已经被验证过的效率机制,系统平移到 AIGC 领域做一遍 ablation。\n\nMMOE 就是来填这个空白的。\n\n## MMOE 到底装了什么\n\n论文明确不把 MoE 当\"单一插件\"对待。它把\"现代化\"的专家设计拆成四个组件,在 SiT 主干上对照实验:\n\n1. **Routed experts**:经典做法,让每个 token 通过 gating 选择 top-k 个专家。MMOE 关注的是不同 routing 策略在 DiT 深度上的稳定性。\n2. **Shared + Lightweight experts**:在 routed 之外加一条\"公共通路\",处理那些 routing 不大能稳定分类的特征。Lightweight experts 是更便宜的共享变体,减少参数膨胀带来的负担。\n3. **Gate-residual routing**:借鉴 LLM 里的残差 gate 思想,把 gate 输出和主干信号相加,而不是替换。这样 routing 选择不会\"覆盖\"掉已经被前一层处理好的特征,收敛更稳。\n4. **Attention-residual information reuse**:在 attention 残差流里复用更早层的路由信号。这条直接拉到了 Transformer 残差设计层面,意味着不同 block 之间的专家调度可以协同,而非各自为战。\n\n这四块不是简单叠加,而是组合起来跑 ablation,看哪一块贡献最大,哪一块是冗余的。\n\n## 实验结果:质量-成本比的胜利\n\n论文核心结论非常硬核:\n\n> **在 matched training + sampling protocol、相同预算下,MMOE 在每个 checkpoint 都达到了比 dense 和 intermediate sparse-expert baseline 更低的 FID。换句话说,它每一步收敛得更快。**\n\n更进一步,在\"都用 MoE\"的稀疏变体内部比较,MMOE 拿到了**最佳的质量-成本平衡**。这意味着 —— 当大家都在比\"我的模型比你的大\"时,MMOE 想说的是\"我的模型比你的便宜,但质量比一样参数量的对头都好\"。\n\nRouting 分析也给出了符合直觉的结论:专家特化在深度方向上稳定存在,lightweight routes 的使用率显著,denoising 的相邻步骤之间路由变化小 —— 说明这套设计不是\"为了 MoE 而 MoE\",而是真的把专家角色做了分工。\n\n## 为什么这件事对产业有意义\n\n把视野拉远一点看,这篇论文在三个方向上都有意义。\n\n**第一,中小实验室能跑了**。8×H100、batch 256、400k 步这个量级,基本就是任何一家有训练预算的中型团队能凑出来的配置。过去 AIGC 论文动辄\"训练消耗 1024 张 H100 几周\",绝大多数从业者只能\"读论文\",没法\"复现\"。MMOE 的预算设定把研究-复现之间的鸿沟压了下来。\n\n**第二,把\"参数堆叠\"的叙事松动了一点**。当下 AIGC 圈子有个隐含共识:模型质量 ≈ 总参数。如果 MMOE 路线被持续验证,那么\"用更聪明的方法,做出质量-成本比更好的扩散模型\"就会变成一个值得押注的方向——相当于 AIGC 版的 LLM-MoE 故事。\n\n**第三,对视频 DiT 是一利好**。Seedance、Veo、Sora 这类视频扩散 Transformer 的算力消耗是文本扩散的几十倍。如果 MMOE 类设计能搬到视频 DiT,推理成本会有显著下降,意味着 B 端可用性和 C 端实时性都能往前走一步。\n\n## 个人评论:这篇论文的真正姿态\n\n我不太想给这篇论文贴上\"颠覆性\"的标签。扩散 Transformer 上 MoE 已经不是什么新鲜事,MMOE 也不是第一个这么做的。但它**难得地把 LLM 那套效率工具箱的每一项都做了 ablation,给出了可复现的实验设定和明确的结论**。在 AIGC 越来越依赖\"砸资源\"的环境里,这种\"克制感\"本身就是一种姿态。\n\n**所以呢**:如果你是一家做图像生成或视频生成的团队,过去默认\"参数 = 竞争力\",MMOE 提醒你还有一条路 —— 把 LLM 已经被验证的效率设计,系统地搬到自己的 DiT 上,做出质量-成本比更好的模型。这个方向如果在更大规模(更多 H100、更长训练)下被验证有效,可能会改变接下来 12-18 个月 AIGC 基础模型的投入逻辑。\n\n数据来源:arXiv:2607.24665,《MMOE: Modernizing Diffusion Transformers with Efficient Expert Design》submitted 27 Jul 2026。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.24665","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"e6ed1c53-e13d-434b-85dc-8b290124da15","en","MMOE brings LLM-style efficient MoE to diffusion transformers","arXiv:2607.24665 introduces MMOE, which systematically brings routed experts, shared\u002Flightweight experts, gate-residual routing, and attention-residual information reuse onto SiT-style diffusion transformers. On a single 8×H100 node, it hits lower FID at every checkpoint than dense baselines and the best quality-cost ratio among sparse variants — showing that AIGC foundation models can follow LLMs' balanced-scaling path.","On July 27, 2026, arXiv posted a paper from Xuelong Li's team at Beihang — \"MMOE: Modernizing Diffusion Transformers with Efficient Expert Design\" (arXiv:2607.24665). It doesn't grab the spotlight like a model release, but it surfaces a question the AIGC community has quietly been avoiding: **does a Diffusion Transformer really need to \"pile up parameters\" to get stronger?**\n\n## The Core in One Sentence\n\nMMOE is not as simple as \"slapping a MoE layer onto DiT\". What it does is systematically port the *entire* \"efficiency toolbox\" that LLMs used to scale over the past few years — routed experts, shared\u002Flightweight experts, gate-residual routing, attention-residual information reuse — onto SiT-style diffusion transformers, and run ablations: when you add one more efficiency component, how does the convergence curve change, how does FID change, and what is the qualitative cost?\n\nThe experimental budget is deliberately modest: **a single 8×H100 node, batch size 256, 400k steps**. This is a setup any mid-sized lab can afford — not a resource play that only frontier labs can mount.\n\n## The Road LLMs Already Walked, That DiT Is Only Now Paving\n\nLet's set the scene. LLMs scaled sustainably not because everyone kept \"adding parameters\", but because MoE decoupled \"total parameters\" from \"activated parameters\" — the total param count can be huge, but each token activates only a small slice, keeping compute in check. At the same time, design choices like mixed routed + shared experts, gate residuals, and attention residual information reuse — these \"small switches\" — collectively let LLM training converge stably without blowing up.\n\nBut on the DiT side, how has it gone? In the past two years, whether for image generation DiT or video DiT, once MoE was bolted on, the inertial move has been: **first ramp up total parameters, then bump the sparsity ratio**. Total params inflate fast, quality does go up, but single-step convergence curves among sparse variants keep beating each other, and the quality-cost trade-off has never been pinned down. In short, everyone has been playing the \"parameter war\", and nobody has carefully ported the LLM's already-validated efficiency mechanisms into AIGC and run a proper ablation.\n\nMMOE exists to fill that gap.\n\n## What MMOE Actually Installs\n\nThe paper explicitly refuses to treat MoE as a \"single plug-in\". It splits the \"modernized\" expert design into four components and runs controlled experiments on the SiT backbone:\n\n1. **Routed experts**: the classical move — let each token's gating pick a top-k of experts. MMOE focuses on the *stability* of different routing strategies across DiT depth.\n2. **Shared + Lightweight experts**: alongside routing, add a \"common pathway\" for features that routing cannot stably categorize. Lightweight experts are cheaper shared variants that cut the param-inflation burden.\n3. **Gate-residual routing**: borrow LLM's residual-gate idea — add the gate's output to the trunk signal instead of replacing it. This way routing choices don't \"overwrite\" features the previous layer already polished, making convergence steadier.\n4. **Attention-residual information reuse**: inside the attention residual stream, reuse routing signals from earlier layers. This pulls up to the level of Transformer residual design itself, meaning expert scheduling across blocks can coordinate instead of operating in silos.\n\nThese four pieces aren't simply stacked — they are ablated in combination to see which one contributes most and which one is redundant.\n\n## Experimental Result: A Quality-Cost Win\n\nThe paper's core claim is unambiguous:\n\n> **Under matched training + sampling protocol and identical budget, MMOE achieves lower FID at every checkpoint than both dense and intermediate sparse-expert baselines. In other words, it converges faster per training step.**\n\nFurther, when comparing *within* the sparse variants (everybody using MoE), MMOE achieves the **best quality-cost balance**. That means — while the community keeps arguing \"my model is bigger than yours\", MMOE says: \"my model is cheaper than yours, and the quality beats competitors with the same param count\".\n\nRouting analysis also reaches intuitive conclusions: expert specialization is stable across depth, lightweight routes see substantial use, and routing changes modestly across adjacent denoising steps. The design is not \"MoE for the sake of MoE\" — it really does assign different roles to different experts.\n\n## Why This Matters for the Industry\n\nZooming out, this paper matters on three axes.\n\n**First, mid-sized labs can now run it.** An 8×H100, batch 256, 400k-step budget is roughly what any mid-sized team with a training budget can assemble. In the recent past, AIGC papers routinely announced \"training consumed 1024 H100s for weeks\"; most practitioners could only \"read the paper\", not \"reproduce it\". MMOE's budget settings compress the research-reproduction gap.\n\n**Second, it loosens the \"parameter stacking\" narrative.** Today's AIGC community has an implicit consensus: model quality ≈ total parameters. If the MMOE line continues to validate, then \"smarter methods that yield better quality-cost ratios\" becomes a direction worth betting on — the AIGC counterpart of the LLM-MoE story.\n\n**Third, it's a tailwind for video DiT.** Compute spend for video diffusion Transformers like Seedance, Veo, and Sora is dozens of times that of text-level diffusion. If MMOE-class designs can transfer to video DiT, inference costs drop materially, which advances both B-end usability and C-end real-time viability.\n\n## Personal Take: What This Paper's Posture Really Says\n\nI'm reluctant to call this paper \"disruptive\". Putting MoE on diffusion transformers is not new, and MMOE isn't the first to do it. But it **does something rare**: it runs an ablation across every component of the LLM efficiency toolbox, with a reproducible setup and clear conclusions. In an AIGC environment increasingly dependent on \"throw resources at it\", that sense of restraint is itself a posture.\n\n**So what**: if you run an image or video generation team and have been defaulting to \"params = competitiveness\", MMOE is your reminder that there is another path — systemically port LLM's proven efficiency designs onto your DiT, and ship models with better quality-cost ratios. If this direction gets validated at larger scale (more H100s, longer training), it could reshape the AIGC foundation-model investment logic over the next 12-18 months.\n\nSource: arXiv:2607.24665, \"MMOE: Modernizing Diffusion Transformers with Efficient Expert Design\" submitted 27 Jul 2026.","mmoe-diffusion-transformer-efficient-experts","2026-08-02T09:00:00Z","2026-08-02T04:03:58.810347Z","2026-08-02T04:03:58.810356Z",true,"agent",162,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"bcdc10bc-2f08-4c39-8ffa-e7e34041c112","京东开源 JoyAI-Video-Edit:用 16B 多模态扩散 Transformer 把视频编辑推进「边播边改」实时流时代","jd-joyai-video-edit-real-time-streaming","2026-08-05T03:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"028e11f1-c29d-47dd-9c66-d4d90bcc4a26","MMOE 之外:AIGC 团队重新算账,单卡 8×H100 也能跑赢参数堆叠","mmoe-diffusion-transformer-efficient-experts-reproducibility-budget","2026-08-02T08:30:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"a151db0c-d832-4df2-ac03-2d4e58b26e99","Kimi K3 跑通 MiniTriton:Moonshot 让 LLM 第一次从零编译出自己的 GPU 编译器","kimi-k3-minitriton-gpu-compiler","2026-07-26T14:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"a07a31d6-3a63-4452-b898-02a6e575682e","Vera：Netflix 把视频编辑拆成编辑层 + 原视频","vera-netflix-caltech-mixture-transformers-edit","2026-06-24T04:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"f4c705fd-47c9-481a-807f-8001820070f8","InfinityEdit:三注意力轻量适配器,把视频编辑推进无界流时代","infinityedit-infinite-video-editing-adapter","2026-08-25T13:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"5ff06769-2251-40fd-a672-f394a1f68965","十人合影谁是谁:腾讯混元 WithEveryone 给群像生成装上身份锚点","witheveryone-group-image-identity-grounding","2026-08-24T13:00:00+00:00"]