[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-meta-expert-cross-layer-moe-expert-sharing":3,"topics-all":38,"news-related-4c6a1f3f-c848-4259-bc88-8672ae5cba0f":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4c6a1f3f-c848-4259-bc88-8672ae5cba0f","MoE 专家跨层共享：省两成参数，残差变双向","清华与上海 AI Lab 论文提出 Meta Expert：让 MoE 专家跨层共享。激活数与专家粒度不变时，用原版 70–80% 的专家即可匹配同等交叉熵，最优配置合计省下 20–30% 专家与 15–25% 参数，并给出基于线性 CKA 与谱基尼系数的冗余度量。","MoE 架构里有一个很少被摆上台面的默认假设：每个专家属于它所在的那一层。路由器在每层内部挑专家，层与层之间的专家池彼此独立。清华与上海 AI Lab 的这篇论文把这个假设拆开看——不同层的专家在拓扑上独立，存储内容却高度相似。\n\n## 先造一把能横向比的尺子\n\n论文先造尺子再谈优化。作者用 linear-CKA 衡量专家两两相似度：给每个专家喂同一批探针 token，取响应矩阵去均值后算协方差，在协方差特征空间里做余弦相似度，得到专家间的 Gram 矩阵 K。K 半正定，它的特征值谱把相似能量摊到各个主方向上，论文用这个谱的基尼系数 ρ 作最终指标：ρ=0 表示能量均匀、专家完全互补，ρ→1 表示能量挤在少数方向、冗余严重。一个漂亮的性质是，若谱是均匀的 rank-k，基尼系数恰好等于 1−k\u002FE，也就是说 ρ 直接等于\"塌缩掉的专家方向占比\"；又因为是比值，跨专家数、跨深度、跨训练预算都能横向比。\n\n尺子量出的直接结论：Meta Expert 占比越高，损失和冗余都越低。反直觉的是，给 Local Expert 更高激活比例确实压冗余，代价是损失变差。论文的解释是，Local Expert 只在一层里被激活，更难训充分，训不足才导致冗余。\n\n## 省下多少\n\n扩展实验是四级深度阶梯，L∈{12,16,20,24} 的 Qwen3.5 风格纯文本 MoE，从零训练在 H200 上。每层 128 个路由专家加 1 个门控共享专家，top-8 路由，注意力按 3 层 GDN 线性注意力加 1 层全注意力的 period-4 交替，隐藏维度从 768 加到 1536，用 Muon 优化器和 WSD 调度。token 预算从 5.31B 到 34.92B，训练算力跨 7.135×10^18 到 2.413×10^20 FLOPs。\n\n在专家总数与激活总数锁死、FLOPs 基本对齐的条件下，共享池版本用原版 70–80% 的专家就能匹配同等交叉熵，参数口径是 75–90%。合并各项消融的最优配置后，相对 Vanilla 省下约 20–30% 的专家、15–25% 的参数，且模型越深越大省得越多。\n\n## 拓扑不是随手定的\n\n论文把共享池的形状也扫了一遍。让某些层只用 Local Expert、把 Meta Expert 按 2\u002F4\u002F8 个子池分组循环访问，一般都会加重损失；反过来，轻度干预反而略好。最终在每个算力档位都最优的组合是：首尾各两层留给 Local Expert，加上 Meta Expert 分成 2 个交错子池。论文由此推断，最浅和最深的层功能分化最明显，彼此几乎不冗余——这也解释了为什么边界层必须保留专属专家。纯 Meta 配置在损失上略好，但训练不稳，最终采用的基线是 75% 比例的 MR75。\n\n## 下游评测并不整齐\n\n这是最该看的一段。论文训了 6B-A1.6B、200B token 的模型，在 MMLU、C-Eval、CMMLU、BBH、TriviaQA、OpenBookQA 上比了五个变体。MR75+Loop2 在 MMLU 上 0.4175、OpenBookQA 上 0.3640、C-Eval 上 0.4487，都高于 Vanilla 的 0.4087、0.3500、0.4354。但 BBH 上 Vanilla 的 0.2972 反倒高于 MR75 的 0.2912，要到 MR75+F2L2 才回到 0.3061；TriviaQA 上 0.2042 出现在 MR75+F2L2+Loop2，Vanilla 是 0.1924。论文自己也说得克制：受限于训练预算，不同变体在不同数据集上各有优势，需要更大规模验证，但引入共享专家的版本整体一致优于 Vanilla。\n\n## 共享把残差路径变成双向\n\n理论部分可能是全文最有分量的一块。Vanilla 分层 MoE 里，第 l 层的专家内容只能被更深的层读到，且因为读写算子不同，到达时已是冻结的混合成分；反向的梯度高速路虽然能把深层信号送回浅层参数，但那是参数级而非激活级的，深层信号会被中间层的训练信号稀释。共享之后，所有层的梯度写进同一份参数 θ，训练结束时 θ 存的是所有层写入的叠加，浅层于是能读到深层写进去的内容——残差通路从单向变双向。\n\n作者在附录里主动收紧了这条论断：共享池的路径长度是 O(1)，与层距无关，这确实优于层式 MoE 那两条都要穿过 |m−l| 个 Jacobian 的旧路径；但单次前向的因果方向并没有改变，浅层读到的是训练期写入的内容，不是同一遍前向里更深层的激活。分析刻画的是访问能力与路径长度，不保证信号一定不消失。\n\n## 已经在 35B 模型上跑起来\n\n论文一作在 [HF 论文页评论区](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2610.27598)说明，这套方法已放大到 Intern-S2-Mobius。该项目的 [GitHub README](https:\u002F\u002Fgithub.com\u002FInternLM\u002FIntern-S2-Mobius)自述：35B 基础模型，Mobius-v0 架构，用全局共享的 Memory 承载知识、多个 Reasoner 反复查询并精炼隐状态，主打 Backward Residual Connection 与 Dynamic Latent Reasoning，从 Qwen3.5-35B 继续预训练再经 SFT 与 RL，技术报告称端到端推理提速近 4 倍，HF 模型卡标注 36B 参数。这里要说清归属：这些数字都来自项目自己的 README 和模型卡，是官方口径，不是第三方复现。\n\n## 所以呢\n\n真正值得关注的不是\"省参数\"这个数字本身——20–30% 的专家节省在任何一次 MoE 效率讨论里都排不上号。值得看的是它顺手推翻了 MoE 一个很少被质疑的默认前提：专家不必是层私有的。如果专家池可以跨层共享，冗余就从\"深度带来的必要成本\"变成\"可以被回收的容量\"。\n\n不过这份结论目前站在有限的地基上。实验主体是从零训练的小规模阶梯，下游评测在 6B-A1.6B 上都算不上整齐，BBH 上甚至出现退化。真正的检验在于共享池到了生产级规模还成不成立——那里专家之间的冗余结构可能完全不同，通信与显存开销也是另一个量级。作者把\"推广到更大模型和更广架构\"列为下一步，而 Intern-S2-Mobius 这个 35B 规模，正好是那个待验证的起点。\n\n论文原文：[arXiv:2610.27598](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.27598)\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.27598","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"54765d20-9be7-498e-9af0-8f4dd2998435","en","Cross-Layer MoE Sharing: Fewer Experts, Two-Way Residuals","Meta Expert shares MoE experts across layers: 70-80% of the original experts match the same cross-entropy, saving 20-30% of the parameter budget.","MoE architectures carry a rarely-examined default assumption: every expert belongs to the layer it sits in. Each router picks experts from within its own layer, and the expert pools of different layers stay mutually independent. A paper from Tsinghua University and Shanghai AI Laboratory takes that assumption apart — experts in different layers are topologically independent, yet the content they store turns out to be highly similar.\n\n## First, a ruler that compares across architectures\n\nThe paper builds a measurement instrument before proposing any optimization. The authors use linear-CKA to quantify pairwise expert similarity: feed every expert the same probe tokens, subtract the sample mean from its response matrix, compute the covariance, then take a cosine similarity in that covariance feature space to obtain a Gram matrix K over the experts. K is positive semi-definite, and its eigenvalue spectrum distributes the similarity energy across principal directions. The paper's headline metric is the Gini coefficient rho of that spectrum: rho=0 means energy is spread evenly and experts are fully complementary, while rho approaching 1 means energy concentrates on a few directions, i.e. heavy redundancy. One elegant property: for a uniform rank-k spectrum the Gini coefficient equals exactly 1-k\u002FE, so rho is literally the fraction of expert directions that have collapsed. Because it is a ratio, it can be compared across expert counts, depths, and training budgets.\n\nThe first thing the ruler shows: the higher the proportion of Meta Experts, the lower both loss and redundancy. Counterintuitively, giving Local Experts a higher activation share does reduce redundancy, but at the cost of worse loss. The paper's explanation is that a Local Expert is activated in only one layer, so it is harder to train to sufficiency — under-training is what produces the redundancy.\n\n## How much is actually saved\n\nThe scaling experiments run a four-rung depth ladder of Qwen3.5-style text-only MoE models at depths L in {12, 16, 20, 24}, trained from scratch on H200 GPUs. Every layer carries 128 routed experts plus one gated shared expert, top-8 routing, attention alternating in a fixed period-4 pattern of three gated-delta-net (GDN) linear-attention layers and one full-attention layer, hidden size growing from 768 to 1536, optimized with Muon under a warmup-stable-decay schedule. Token budgets run from 5.31B to 34.92B and training compute spans 7.135e18 to 2.413e20 FLOPs.\n\nWith the total expert count and total activation count locked — so FLOPs are essentially matched — the shared-pool version matches the same cross-entropy using only 70–80% of the original experts, or 75–90% of the parameters. Combining the best configuration from each ablation, the savings come to roughly 20–30% of experts and 15–25% of parameters relative to Vanilla, and the advantage grows as models get deeper and larger.\n\n## The topology is not an afterthought\n\nThe paper also sweeps the shape of the shared pool. Confining some layers to Local Experts only, or grouping Meta Experts into 2, 4, or 8 sub-pools accessed in a loop, generally degrades loss; mild interventions, by contrast, slightly help. The combination that performs best at every compute budget is: Local Experts reserved for the first two and last two layers, plus Meta Experts partitioned into two interleaved sub-pools. The authors infer from this that the shallowest and deepest layers serve the most differentiated functions and are essentially free of mutual redundancy — which explains why boundary layers must keep dedicated experts. A pure-Meta configuration achieves slightly better loss but is training-unstable, so the adopted baseline is the 75% ratio, called MR75.\n\n## The downstream numbers are not tidy\n\nThis is the section worth reading. The paper trains a 6B-A1.6B model on 200B tokens and compares five variants across MMLU, C-Eval, CMMLU, BBH, TriviaQA, and OpenBookQA. MR75+Loop2 reaches 0.4175 on MMLU, 0.3640 on OpenBookQA, and 0.4487 on C-Eval, all above Vanilla's 0.4087, 0.3500, and 0.4354. But on BBH, Vanilla's 0.2972 is higher than MR75's 0.2912, and it takes MR75+F2L2 to recover to 0.3061; the best TriviaQA result, 0.2042, comes from MR75+F2L2+Loop2 against Vanilla's 0.1924. The paper is notably restrained in its own framing: given the limited training budget, different variants show advantages on different datasets and larger-scale experiments are needed, yet the variants with shared experts consistently outperform Vanilla overall.\n\n## Sharing turns the residual path bidirectional\n\nThe theoretical section may be the most valuable part of the paper. In a Vanilla layer-wise MoE, content written by layer l's experts can only be read by deeper layers, and because the read operator differs from the write operator, it arrives as a frozen blended component. The backward gradient highway can carry deep-layer signal to shallow parameters, but that is parameter-level rather than activation-level, and the deep signal gets diluted by the training signals of all intermediate layers. With sharing, every layer's gradient writes into one and the same parameter block theta, so after training theta holds a superposition of all layers' writes — a shallow layer can now read content that deeper layers wrote. The residual path becomes bidirectional instead of one-way.\n\nThe authors tighten this claim in the appendix themselves: the path through the shared pool has O(1) length independent of layer distance, which does beat both pre-existing layer-wise routes that pass through |m-l| Jacobians. But the causal direction of a single forward pass is unchanged — what a shallow layer accesses is content written into the shared pool during training, not the activations of deeper layers within the same pass. The analysis characterizes access and path length, and does not guarantee against vanishing signals.\n\n## Already running at 35B\n\nThe paper's submitter states on the [HF paper page](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2610.27598) that the method has been scaled up in Intern-S2-Mobius. That project's [GitHub README](https:\u002F\u002Fgithub.com\u002FInternLM\u002FIntern-S2-Mobius) describes: a 35B foundation model on the Mobius-v0 architecture, organizing knowledge into a globally shared Memory while multiple Reasoners iteratively query and refine hidden states, built around Backward Residual Connection and Dynamic Latent Reasoning, continually pre-trained from Qwen3.5-35B and then post-trained with SFT and RL, with nearly 4x end-to-end inference speedup claimed in the technical report; the HF model card lists 36B parameters. Attribution matters here: those numbers all come from the project's own README and model card, so they are the vendor's own figures, not a third-party reproduction.\n\n## So what\n\nWhat is genuinely interesting is not the parameter saving itself — a 20–30% expert reduction would rank nowhere near the top of any MoE efficiency discussion. What matters is that the work quietly overturns one of MoE's rarely questioned premises: experts do not have to be layer-private. If expert pools can be shared across depth, redundancy stops being an unavoidable cost of depth and becomes capacity that can be reclaimed.\n\nStill, the conclusion currently rests on limited ground. The bulk of the experiments is a small from-scratch ladder, the downstream evaluation at 6B-A1.6B is far from tidy, and BBH even shows a regression. The real test is whether the shared pool still holds at production scale, where the redundancy structure between experts may look entirely different and where communication and memory overhead live in a different order of magnitude. The authors list \"extending to larger models and broader architectures\" as future work — and Intern-S2-Mobius at 35B is precisely the starting point that still has to be validated.\n\nPaper: [arXiv:2610.27598](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.27598)\n","meta-expert-cross-layer-moe-expert-sharing","2026-10-25T11:20:00Z","2026-10-25T11:14:55.126781Z","2026-10-25T11:14:55.126791Z",true,"agent",9,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"8c579d8d-4114-4d7d-92c1-22f5fab0537b","连续流语言模型跑到 8B：Lafa 单模型自验证解码提速 2.54 倍","large-flow-language-models-lafa-self-speculative-decoding","2026-10-21T03:09:03+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"b369ce1a-f8a3-4aba-ba78-c73c097a413d","MemT-1.0：记忆变成模型内置能力，128K 历史存储降 66 倍","memt-1-0-memory-native-foundation-model","2026-10-20T11:07:23+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"50846ffe-7171-471c-889c-247a3b86d553","华为InferOpt:逐层旋钮搜着调,KV缓存省64%","inferopt-llm-inference-config-search","2026-10-14T21:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"f1ccbea4-5749-4976-bfbd-9fc835318226","Foil 重构循环 MoE:专家压进单层,循环次数翻八倍","foil-loop-moe-flatten-untie","2026-10-07T19:09:13+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a78b61ba-6899-4b4f-86b0-8d6c1753308b","亚马逊 ALoDLM:扩散语言模型学会按难度分配算力","alodlm-adaptive-looped-diffusion","2026-10-07T13:15:53+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"44b118b2-cb3d-47bb-8f26-bbce777cfb31","1 亿 rollout 背后:Beam 的 RL 训练工厂","beam-rl-factory-100m-rollouts","2026-10-07T07:15:00+00:00"]