[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-moe-overfit-repeated-data-stanford-uw":3,"topics-all":38,"news-related-70529122-522e-405d-9735-fc083706792f":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"70529122-522e-405d-9735-fc083706792f","数据重复 4 倍就开始退化:斯坦福UW团队实测 MoE 比稠密模型更怕数据墙","人类文本即将耗尽,重复训练数据已成行业标准做法。斯坦福与 UW 团队实测 80M-1B 激活参数模型发现:MoE 在数据重复 4 倍时就开始退化、32 倍后被稠密模型反超,且稀疏度越高退化越狠;dropout 等掩码正则能部分挽救。","大模型训练正在撞上「数据墙」:人类书写的文本就那么多,把训练数据重复几遍用,已经从权宜之计变成行业标准做法。与此同时,主流大模型几乎清一色转向 MoE(混合专家)架构,用稀疏激活换算力效率。这两个趋势撞在一起会发生什么?此前几乎没人系统研究过。9 月 10 日,斯坦福与华盛顿大学团队在 arXiv:2609.11917 发表论文《Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data》,作者名单包括 Jure Leskovec、Percy Liang 和 Luke Zettlemoyer,结论相当扎心:MoE 比稠密模型更怕重复数据。\n\n## 4 倍重复就开始退化\n\n团队做了 compute-matched 网格实验:80M、200M、1B 激活参数(总参最高 8.5B),专家数 8 到 128,粒度 1\u002F2 到 1\u002F32,重复率 R 从 1 扫到 1024,数据覆盖 OLMoE 数据混合的四个域——网页(DCLM)、代码(StarCoder)、科学文本(peS2o)和维基百科。\n\n核心数字:80M 稠密模型重复数据 8 倍几乎无损,同规模 MoE 重复 4 倍就开始退化,到 32 倍时被稠密模型反超,全独特数据下的架构优势荡然无存。更关键的是,退化程度跟着总参数走而不是激活参数:稀疏度越高,崩得越狠。这个模式在四个数据域和不同混合配比下高度一致,质量过滤也几乎改变不了什么。\n\n## 机制:路由早固化,专家过特化\n\n为什么会这样?机制分析给出了两条证据。其一,MoE 的路由器在训练早期就「固化」:第 400 步(约训练进度 10%)路由稳定性已到 60%,训练结束时超过 95%——每个专家在绝大部分训练里反复更新同一小撮 token,数据一重复,它看到的就是同一块数据的更多遍。其二,专家 knockout 实验:重复率从 1 拉到 32,80M 下 128 专家、1\u002F4 粒度配置的专家缺失代价涨 2.3 倍,专家特化与过拟合明显相关。\n\n## 解药存在,但不完美\n\n正则化实验给出活路:dropout、FFN 输出掩码、专家 dropout、专家输出掩码这类「丢弃参数输出」的方法都有效,配合强掩码正则,MoE 在重复超过 64 倍时仍能跑赢稠密模型。而 weight decay、梯度裁剪、router jitter 几乎没有可测影响。论文也很诚实:没有任何方法能完全追平全独特数据。另一个实用发现:把重复域混进足量未重复数据,未重复部分会起正则作用。\n\n## 所以呢\n\n这篇论文的真正贡献,是把行业默认的架构选择变成了条件命题:稀疏化的红利,取决于你的 unique token 预算。数据管够时 MoE 是免费午餐,数据见底时它比稠密模型更脆弱。对规划下一代基座模型的团队来说,先数清楚手里还有多少没重复过的数据,再决定上多大的稀疏度,可能比多加几千张卡更值得算清楚。[原文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.11917)\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.11917","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"d1033fe2-f5e0-48e8-9b22-be4c7ed4012e","en","MoE Overfits More to Repeated Data, Stanford-UW Study Finds","Stanford-UW study: MoE models degrade from 4x data repetition and trail dense models at 32x; masking regularization offers partial rescue.","Large-scale model training is running into a data wall: the supply of human-written text is finite, and repeating training data has shifted from a stopgap to standard industry practice. Meanwhile, frontier models have almost uniformly moved to Mixture-of-Experts (MoE) architectures, trading sparse activation for compute efficiency. What happens when these two trends collide? Until now, almost nobody had studied it systematically. On September 10, a Stanford–University of Washington team posted arXiv:2609.11917, \"Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data,\" authored by a group including Jure Leskovec, Percy Liang, and Luke Zettlemoyer. The verdict is uncomfortable: MoEs are more fragile under repeated data than dense models.\n\n## Degradation starts at 4x repetition\n\nThe team ran compute-matched grid experiments: 80M, 200M, and 1B active parameters (up to 8.5B total), expert counts from 8 to 128, granularities from 1\u002F2 to 1\u002F32, repetition rates swept from 1 to 1024, across the four domains of the OLMoE data mix — web crawl (DCLM), code (StarCoder), scientific text (peS2o), and Wikipedia.\n\nThe core numbers: 80M dense models can repeat data 8x with minimal degradation, while same-scale MoEs start suffering at 4x and are overtaken by dense models at 32x, wiping out the advantage they hold on all-unique data. Crucially, degradation tracks total parameters, not active ones: the sparser the model, the harder it falls. The pattern holds across all four data domains and mixture ratios, and quality filtering barely changes it.\n\n## Mechanism: early router ossification, expert over-specialization\n\nWhy? The mechanistic analysis offers two lines of evidence. First, MoE routers \"ossify\" early in training: routing stability reaches 60% by step 400 (about 10% of training) and exceeds 95% by the end — so each expert keeps updating on the same small shard of tokens for most of training, and once data is repeated, it sees that same shard even more often. Second, expert-knockout experiments: raising repetition from 1 to 32 increases knockout cost 2.3x for the 80M, 128-expert, 1\u002F4-granularity configuration, and expert specialization correlates clearly with overfitting.\n\n## Remedies exist, but none are complete\n\nThe regularization experiments offer a lifeline: methods that drop parameter outputs — dropout, FFN output masking, expert dropout, expert output masking — all work, and with strong masking-based regularization MoEs still beat dense models at more than 64x repetition. Weight decay, gradient clipping, and router jitter show no measurable effect. The paper is also honest that no method fully matches all-unique data. One practical finding: mixing a repeated domain into enough non-repeated data lets the unique portion act as a regularizer.\n\n## So what\n\nThe real contribution is not the headline \"MoEs overfit\" — it is turning an industry-default architecture choice into a conditional statement: the dividend of sparsity depends on your unique-token budget. With abundant data, MoE is a free lunch; when data runs dry, it is more fragile than dense models. For teams planning the next base model, counting how much truly unique data is left before deciding how much sparsity to build in may be worth more than a few thousand extra GPUs. [Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.11917)\n","moe-overfit-repeated-data-stanford-uw","2026-09-13T19:20:00Z","2026-09-13T19:10:12.627263Z","2026-09-13T19:10:12.627279Z",true,"agent",87,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"2266cea6-06f1-4932-8905-1bc3f2e5a8c0","Meta FAIR 字节蒸馏研究:End-Of-Token 渐近反超 token 蒸馏 4%,数据只需 1\u002F6","meta-fair-byte-distillation-token-ceiling-2026-09","2026-09-15T02:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"05f36392-7a87-4eac-af6b-d2606c4add83","AMD 把 Instella-MoE 摊在桌面:16B 总量、2.8B 激活的「全开源 MoE」想证明什么","amd-instella-moe-fully-open-moe-16b-a3b","2026-08-12T10:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"cb7fb8b3-5862-4cba-adab-c4794e989966","图灵奖得主 Pearl 长访谈：LLM 能讲因果只是因为人类替它爬过了因果阶梯","judah-pearl-llm-causal-ladder-agi","2026-07-31T07:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"f8436dd3-d6fc-4ea7-9f2e-1086026c11d0","Transformer 的几何之眼：arXiv 2607.17146 把注意力炼成薛定谔桥，把 SGD 写成伊藤扩散","transformer-geometry-schrodinger-bridge","2026-07-23T12:10:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"b5638cab-a4d6-44ac-9230-32ed0a4cba9d","ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文","armt-associative-recurrent-memory-transformer","2026-07-23T00:00:00+00:00"]