[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-amd-instella-moe-fully-open-moe-16b-a3b":3,"topics-all":38,"news-related-05f36392-7a87-4eac-af6b-d2606c4add83":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"05f36392-7a87-4eac-af6b-d2606c4add83","AMD 把 Instella-MoE 摊在桌面:16B 总量、2.8B 激活的「全开源 MoE」想证明什么","AMD 发布 Instella-MoE-16B-A3B:160 亿总参数、每个 token 仅激活 28 亿参数的「全开源」MoE,采用 Gated MLA 与 FarSkip-Collective 两项自研优化,在 MI300X\u002FMI325 全栈上训出 64K 长上下文 + 完整 RLHF,Base 平均 76.7、Think 模型 IFEval 83.70。它真正想卖的不是一个新模型,而是 AMD 第一次从 GPU 到 RLHF 的「全栈主权」。","# AMD 把 Instella-MoE 摊在桌面：16B 总量、2.8B 激活的「全开源 MoE」想证明什么\n\n## 事件\n\nAMD 7 月 24 日通过 ROCm Blogs 发布 [Instella-MoE-16B-A3B](https:\u002F\u002Frocm.blogs.amd.com\u002Fartificial-intelligence\u002Finstella-moe\u002FREADME.html),一款「完全开源」的 Mixture-of-Experts 语言模型。它有 160 亿总参数,每个 token 仅激活 28 亿参数,采用 27 层 decoder-only 架构、2 个共享 expert 加 64 选 6 的 routed expert 结构。这不是普通的研究 demo:AMD 同时放出了从预训练、Mid-train、长上下文扩展到 SFT、DPO、强化学习的**完整 6 阶段 checkpoint**,以及对应的训练配方、数据配比和代码。这正是「全开源」和「只开源权重」之间的关键差距。\n\n## 两条架构创新:Gated MLA 与 FarSkip-Collective\n\nInstella-MoE 没有照搬 DeepSeek-V3 的 MLA,而是在 MLA 上加了一个**轻量级 learned output gate**——这就是论文里所说的 Gated Multi-head Latent Attention。做法不复杂:每个 Gated MLA 层先做一个独立的线性投影得到一个 input-conditioned gate,然后在 MLA 输出与最终 output projection 之间做逐元素相乘。这个 gate 是数据依赖的非线性,可以让模型**有选择地弱化低价值的 attention 响应**。AMD 强调「以适度的成本换取了更大的表达力」——也就是说,Gated MLA 不是为 benchmark 数字服务的花活,而是给 MoE 模型本身的稀疏性留出一个对 attention 输出做 soft gating 的口子。\n\n另一条更偏系统的优化是 **FarSkip-Collective**:它改造了 MoE 中常用的 all-to-all expert parallel,把「过时且不完整的激活」提前传进下一层 attention 和 MoE,从而把通信和计算 overlap。AMD 报告在预训练阶段通信 overlap 带来了 **12.7%** 的训练提速,在推理侧配合 SGLang 与 expert parallelism 后,TTFT(Time to First Token)最高降低 **39.2%**。这意味着 Instella-MoE 不只是一个能跑通的模型,还是一个**为 AMD 硬件通信栈专门调过**的模型。\n\n## 训练管线:64K 长上下文、1400 步 GRPO、MOPD 蒸馏\n\n长上下文阶段把窗口从 4K 扩到 64K,使用了 YaRN 风格的 RoPE theta 调整,并在 packed document 之间加上了 document mask 以防 attention 泄漏。两阶段 long-context 训练分别针对通用长文档和 math\u002Fcode\u002Freasoning 长文本,在最终的 Base checkpoint 上,HELMET 平均 41.5、RULER 平均 79.4。\n\n后训练分三步:\n- **SFT**:在 Dolci-Think-SFT-7B 之上叠 Nemotron-Cascade-2-SFT-Data 与 Nemotron-SFT-Competitive-Programming-v2 强化 STEM 与推理;末期引入 feedback-driven data curation,用 512K 精选样本替换均匀采样。\n- **DPO**:AMD 在初步实验里发现 DPO 直接作用于 MoE 会引发性能退化,他们的解释是 load-balancing 目标会让 router 的 affinity 快速变化,因此在 DPO 阶段**禁用了 router bias 更新和辅助 load-balancing 损失**——这是工程上比较有意思的细节。\n- **RL**:用 IF-RLVR 子集跑 1400 步 GRPO,叠加 zero-gradient signal filtering、active sampling、token-level loss、no KL、clip-higher、no std normalization 与 Rollout Routing Replay (R3);最后用 Multi-Teacher On-Policy Distillation(MOPD)把 IF expert 的指令跟随能力蒸馏回 DPO 模型。\n\n## 成绩单:在 2.8B 激活预算内追平 4B 量级\n\n- **Base 模型**平均分 76.7,在「全开源」阵营里最强,明显高于 SmolLM3-3B-Base(70.5)、OLMo-3-7B(70.1)、OLMoE-1B-7B(61.9);WinoGrande 86.5 排名第一、HumanEval+ 65.7,跨知识\u002F推理\u002F数学\u002F代码分布均衡。\n- **Think 模型**(SFT → DPO → RL → MOPD 蒸馏)平均 73.22,在 Olmo3-7B-Think(71.97)、Gemma-4-E4B-think(70.47)、Qwen3.5-4B(69.73)之上;IFEval 从 DPO 的 77.08 抬到 83.70,AGIEval 82.50、AIME25 73.40。\n\n换句话说,**16B 总量 \u002F 2.8B 激活**这个预算,AMD 跑出了在「全开源」和「开源权重」两个圈子里都不输的分数。这背后是 MoE 稀疏激活 + Gated MLA 表达力 + AMD 全栈通信优化的合力。\n\n## 评论:AMD 想要的不是 benchmark,是「全栈主权」\n\n把这件事拆开看,有几个值得拆的层:\n\n**第一,AMD 这次卖的不是模型,是栈。** 从 Instinct MI300X\u002FMI325 GPU、ROCm 软件栈、Primus 训练框架、Miles RL 框架、SGLang 推理,到最终模型 weights——AMD 第一次公开展示「**在自家长链路里从头到尾训出一个 MoE LLM**」的能力。对比 NVIDIA 那一边的 NeMo \u002F Megatron \u002F TensorRT-LLM 体系,这是 AMD 在 LLM 时代的第一次完整亮相。\n\n**第二,「全开源」和「只开源权重」的鸿沟,AMD 用 checkpoint 阶梯填上了一半。** Qwen3.8-Max、Llama 4、DeepSeek V4 这些「开源权重」阵营,通常只放最终 SFT 或 Instruct checkpoint,中间过程基本看不到。Instella-MoE 把预训练、Mid-train、长上下文、SFT、DPO、RL 全 6 个阶段的 checkpoint 都开放,意味着研究者可以**逐阶段复现**——这才是「学术可复现」的真正定义,也解释了为什么 AMD 把它定位为「state-of-the-art fully open」而不是简单的「open-source」。\n\n**第三,Gated MLA 和 FarSkip-Collective 是工程级而非架构级的创新。** 它们的 paper-grade 不算顶尖,但**对 AMD 自己的硬件栈来说,是关键拼图**——AMD 的 GPU 互联带宽弱于 NVLink,FarSkip-Collective 这种「用不完整 activation 提前 overlap」的手法,本质是用算法补带宽。这也是为什么 AMD 把 12.7% 训练加速、39.2% TTFT 下降当成重点宣传:这是他们能给客户讲的「跑在我们 GPU 上更快」的硬指标。\n\n**第四,这是一个**「MoE 工程民主化」**的样本。** MoE 不再是只有头部玩家玩得起的奢侈品:16B 总量 + 2.8B 激活 + 64K context + 完整 RLHF 管线,跑在消费级 GPU 集群上,而不是非要 1000 张 H100。AMD 等于在说:**只要栈足够透明,中等规模 MoE 也能跑通 RL 蒸馏。**\n\n## 所以呢\n\n如果你是 LLM 从业者,Instella-MoE 的真正价值在「**checkpoint 阶梯 + 完整训练配方**」——你可以拿它的 Mid-train 自己做下游微调,也可以复现它的 GRPO 改造。如果你是硬件买家,它的 FarSkip-Collective 和 Gated MLA 提供了**「AMD GPU 跑 MoE 也不输」的实证**。如果你是开源生态观察者,这次的「**全开源 6 阶段**」是一个值得 Meta、DeepSeek、Qwen 跟进的标准。\n\n最后一个小提醒:AMD 给 Instella-MoE 用的是 **Research RAIL 许可证**——它只供学术研究,不接受商业使用。所以它的「商业影响力」短期内受限,但**学术复现价值**反而更高。\n\n参考:\n- AMD ROCm Blogs: [Introducing Instella-MoE](https:\u002F\u002Frocm.blogs.amd.com\u002Fartificial-intelligence\u002Finstella-moe\u002FREADME.html)\n- 配套训练框架 Primus: [ROCm Blogs](https:\u002F\u002Frocm.blogs.amd.com\u002Fsoftware-tools-optimization\u002Fprimus\u002FREADME.html)\n- Miles RL 框架(与 LMSYS 合作): [LMSYS Blog](https:\u002F\u002Fwww.lmsys.org\u002Fblog\u002F2026-03-17-rocm-miles-rl-amd\u002F)\n- FarSkip-Collective 论文: [OpenReview @ MLSys 2026](https:\u002F\u002Fopenreview.net\u002Fforum?id=ruOpvLzsGV)\n- Hugging Face 模型集合: [amd\u002Finstella-moe](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Famd\u002Finstella-moe)","https:\u002F\u002Frocm.blogs.amd.com\u002Fartificial-intelligence\u002Finstella-moe\u002FREADME.html","09817576-1b8d-491e-b843-2913b7bcbe49",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"88b3b348-cc04-4750-a8d9-2b69dbfc7652","en","AMD's Instella-MoE: 16B open MoE with 2.8B active","AMD released Instella-MoE-16B-A3B: a 'fully open' Mixture-of-Experts LLM with 16B total parameters but only 2.8B active per token, featuring two custom optimizations — Gated MLA and FarSkip-Collective — and trained end-to-end on MI300X\u002FMI325 GPUs with a 64K long-context + complete RLHF pipeline. The Base model averages 76.7, the Think model reaches 83.70 on IFEval. What AMD is really selling here is not a new model but its first 'full-stack sovereignty,' from silicon all the way to RLHF.","# AMD Lays Instella-MoE on the Table: A 16B \u002F 2.8B-Active \"Fully Open\" MoE That Wants to Prove a Point\n\n## What happened\n\nOn July 24, AMD published [Instella-MoE-16B-A3B](https:\u002F\u002Frocm.blogs.amd.com\u002Fartificial-intelligence\u002Finstella-moe\u002FREADME.html) on ROCm Blogs — a \"fully open\" Mixture-of-Experts language model with 16 billion total parameters but only 2.8 billion active per token. The architecture is a 27-layer decoder-only MoE with 2 shared experts plus a 64-routes-6 routed-experts design. This is not a research demo: AMD shipped **all six training-stage checkpoints** (pretrain, mid-train, long-context, SFT, DPO, RL), plus the full training recipe, data mixtures, and code. That is the real distance between \"fully open\" and \"open weights only.\"\n\n## Two architectural touches: Gated MLA and FarSkip-Collective\n\nInstella-MoE doesn't copy DeepSeek-V3's MLA verbatim. Instead it adds a **lightweight learned output gate** on top of MLA — what AMD calls Gated Multi-head Latent Attention. Each Gated MLA layer runs a dedicated linear projection to compute an input-conditioned gate, then multiplies it into the MLA output before the final projection. This data-dependent nonlinearity lets the model **selectively attenuate low-utility attention responses**, gaining expressivity \"at modest cost.\" It's a small change, but it gives the MoE's sparse activations a soft gate over the attention output that the vanilla MLA does not provide.\n\nThe second innovation, **FarSkip-Collective**, is more systems-flavored. It modifies the standard MoE all-to-all expert-parallel pattern by passing outdated and partial activations ahead into the next attention and MoE layers, overlapping communication with computation. AMD reports **12.7% pretraining speedup** from this overlap, and — when the model is served through SGLang with expert parallelism — up to **39.2% reduction in Time-to-First-Token (TTFT)**. Instella-MoE is not just a model that runs; it is a model tuned for **AMD's hardware communication stack**.\n\n## Training pipeline: 64K long context, 1,400 GRPO steps, MOPD distillation\n\nThe long-context stage extends the window from 4K to 64K tokens using a YaRN-style RoPE theta adjustment and document masking on packed sequences. The two-phase recipe first teaches general long-context attention, then sharpens math \u002F code \u002F reasoning long-text performance. The Base checkpoint lands at a HELMET average of 41.5 and a RULER average of 79.4.\n\nPost-training has three phases:\n\n- **SFT.** Built on Dolci-Think-SFT-7B, augmented with Nemotron-Cascade-2-SFT-Data and Nemotron-SFT-Competitive-Programming-v2 to strengthen STEM and reasoning. The final SFT stage uses a feedback-driven data curation pipeline that generates errors with an intermediate checkpoint, has a strong judge analyze them, retrieves a weighted 512K-example targeted mixture, and combines it with uniformly-sampled examples for breadth.\n- **DPO.** In preliminary experiments, AMD found that applying DPO directly to the MoE caused performance regression. Their hypothesis: the load-balancing objective rapidly shifts the experts' router affinity. The fix: **disable router bias updates and the auxiliary load-balancing loss during DPO**. That's an interesting engineering detail worth flagging.\n- **RL.** 1,400 GRPO steps on the IF-RLVR subset of Dolci-Think-RL-7B, with asynchronous training and 16K-token responses. AMD layers in zero-gradient signal filtering, active sampling, token-level loss, no KL, clip-higher, no std normalization, and Rollout Routing Replay (R3). Then **Multi-Teacher On-Policy Distillation (MOPD)** distills the instruction-following expert back into the DPO model.\n\n## The scoreboard: matching 4B-scale at a 2.8B-active budget\n\n- **Base model** averages 76.7 — strongest among fully open models, ahead of SmolLM3-3B-Base (70.5), OLMo-3-7B (70.1), OLMoE-1B-7B (61.9); WinoGrande 86.5 leads the field; HumanEval+ 65.7; balanced across knowledge, reasoning, math, and code.\n- **Think model** (SFT → DPO → RL → MOPD) averages 73.22, ahead of Olmo3-7B-Think (71.97), Gemma-4-E4B-think (70.47), and Qwen3.5-4B (69.73). IFEval jumps from 77.08 (DPO) to 83.70; AGIEval 82.50; AIME25 73.40.\n\nIn other words, **16B total \u002F 2.8B active** lands in the ballpark of leading 4B-scale models. The combination is MoE sparsity + Gated MLA expressivity + AMD-stack communication optimization.\n\n## Why this matters: AMD is selling the stack, not the benchmark\n\nA few layers worth unpacking:\n\n**First, AMD is not selling a model — it is selling a stack.** Instinct MI300X \u002F MI325 GPUs, ROCm software, Primus training framework, Miles RL framework, SGLang inference, and finally the model weights. This is AMD's first complete demonstration of **end-to-end LLM training on its own long toolchain** — a direct counterpart to NVIDIA's NeMo \u002F Megatron \u002F TensorRT-LLM orbit.\n\n**Second, the gap between \"fully open\" and \"open weights only\" is real, and AMD fills half of it.** Qwen3.8-Max, Llama 4, DeepSeek V4 — the \"open weights\" camp usually ships only the final SFT or Instruct checkpoint; intermediate states are opaque. Instella-MoE ships **all six** checkpoints, which means researchers can **reproduce stage by stage**. That is what \"academically reproducible\" actually means, and it's why AMD uses the label \"state-of-the-art fully open\" rather than just \"open-source.\"\n\n**Third, Gated MLA and FarSkip-Collective are engineering-level, not architecture-level, innovations.** They're not paper-grade breakthroughs. But for **AMD's own hardware stack** they are load-bearing — AMD's GPU interconnect bandwidth trails NVLink, so FarSkip-Collective's \"feed incomplete activations ahead to overlap\" trick is essentially algorithm-as-bandwidth-substitute. The 12.7% training speedup and 39.2% TTFT reduction are exactly the hard numbers AMD needs to tell customers \"our GPUs run MoE faster, too.\"\n\n**Fourth, this is a sample of \"MoE engineering democratization.\"** MoE is no longer a luxury only the frontier labs can afford: 16B total + 2.8B active + 64K context + a complete RLHF pipeline, running on consumer-grade GPU clusters rather than demanding 1,000 H100s. AMD is essentially saying: **with a transparent enough stack, mid-scale MoE can complete the RL distillation loop too.**\n\n## So what\n\nIf you build LLMs, the real value of Instella-MoE is the **checkpoint ladder + complete training recipe** — you can take its Mid-train checkpoint and fine-tune downstream, or reproduce its GRPO modifications. If you're a hardware buyer, FarSkip-Collective and Gated MLA give you **empirical evidence that AMD GPUs run MoE without losing** to the alternative. If you watch the open ecosystem, this \"**fully open six-stage**\" release sets a standard Meta, DeepSeek, and Qwen will be asked to match.\n\nOne caveat: AMD ships Instella-MoE under a **Research RAIL license** — research-only, no commercial use. That limits its short-term commercial reach but actually **raises its academic reproducibility value**.\n\nReferences:\n- AMD ROCm Blogs: [Introducing Instella-MoE](https:\u002F\u002Frocm.blogs.amd.com\u002Fartificial-intelligence\u002Finstella-moe\u002FREADME.html)\n- Primus training framework: [ROCm Blogs](https:\u002F\u002Frocm.blogs.amd.com\u002Fsoftware-tools-optimization\u002Fprimus\u002FREADME.html)\n- Miles RL framework (with LMSYS): [LMSYS Blog](https:\u002F\u002Fwww.lmsys.org\u002Fblog\u002F2026-03-17-rocm-miles-rl-amd\u002F)\n- FarSkip-Collective paper: [OpenReview @ MLSys 2026](https:\u002F\u002Fopenreview.net\u002Fforum?id=ruOpvLzsGV)\n- Hugging Face collection: [amd\u002Finstella-moe](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Famd\u002Finstella-moe)","amd-instella-moe-fully-open-moe-16b-a3b","2026-08-12T10:00:00Z","2026-08-12T12:06:16.558407Z","2026-08-12T12:06:16.558414Z",true,"agent",149,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"176b4807-da61-479f-a514-9381cd13319e","SP3O:3 个锚点修复 PPO critic 的平坦化","sp3o-sparse-critic-supervision","2026-09-17T17:10:01+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"7bae3d71-a5c2-4588-95e7-b5d4b5c7085a","开源模型 4.4 个月追上闭源前沿:Hugging Face 被 NVIDIA 129 亿美元收编","nvidia-acquires-hugging-face-open-source-ai","2026-09-17T08:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"21fe3c11-4ba4-4801-b6fc-60c4ae559dc1","Yandex 逆流开源:35B 参数的 T5 MoE,每个 token 只激活 0.6B","yandex-aliceai-t5-sparse-moe","2026-09-16T19:11:43+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"70529122-522e-405d-9735-fc083706792f","数据重复 4 倍就开始退化:斯坦福UW团队实测 MoE 比稠密模型更怕数据墙","moe-overfit-repeated-data-stanford-uw","2026-09-13T19:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"8e730a3d-439b-45cf-961d-f77cf01469fd","Cohere 开源 218B 翻译专用 MoE:25B 激活,自测评分超 DeepL,2×H100 可部署","cohere-north-small-translate","2026-09-11T19:07:20+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"d41175a7-ad10-4e00-9017-a148fa0a77b3","BenchMIRT 把 LLM 基准拆到单题:Ai2 想让模型排名不再「一张考卷定生死」","ai2-benchmirt-llm-benchmark-audit","2026-09-10T11:05:05+00:00"]