AMD Lays Instella-MoE on the Table: A 16B / 2.8B-Active "Fully Open" MoE That Wants to Prove a Point

What happened

On July 24, AMD published Instella-MoE-16B-A3B on ROCm Blogs — a "fully open" Mixture-of-Experts language model with 16 billion total parameters but only 2.8 billion active per token. The architecture is a 27-layer decoder-only MoE with 2 shared experts plus a 64-routes-6 routed-experts design. This is not a research demo: AMD shipped all six training-stage checkpoints (pretrain, mid-train, long-context, SFT, DPO, RL), plus the full training recipe, data mixtures, and code. That is the real distance between "fully open" and "open weights only."

Two architectural touches: Gated MLA and FarSkip-Collective

Instella-MoE doesn't copy DeepSeek-V3's MLA verbatim. Instead it adds a lightweight learned output gate on top of MLA — what AMD calls Gated Multi-head Latent Attention. Each Gated MLA layer runs a dedicated linear projection to compute an input-conditioned gate, then multiplies it into the MLA output before the final projection. This data-dependent nonlinearity lets the model selectively attenuate low-utility attention responses, gaining expressivity "at modest cost." It's a small change, but it gives the MoE's sparse activations a soft gate over the attention output that the vanilla MLA does not provide.

The second innovation, FarSkip-Collective, is more systems-flavored. It modifies the standard MoE all-to-all expert-parallel pattern by passing outdated and partial activations ahead into the next attention and MoE layers, overlapping communication with computation. AMD reports 12.7% pretraining speedup from this overlap, and — when the model is served through SGLang with expert parallelism — up to 39.2% reduction in Time-to-First-Token (TTFT). Instella-MoE is not just a model that runs; it is a model tuned for AMD's hardware communication stack.

Training pipeline: 64K long context, 1,400 GRPO steps, MOPD distillation

The long-context stage extends the window from 4K to 64K tokens using a YaRN-style RoPE theta adjustment and document masking on packed sequences. The two-phase recipe first teaches general long-context attention, then sharpens math / code / reasoning long-text performance. The Base checkpoint lands at a HELMET average of 41.5 and a RULER average of 79.4.

Post-training has three phases:

  • SFT. Built on Dolci-Think-SFT-7B, augmented with Nemotron-Cascade-2-SFT-Data and Nemotron-SFT-Competitive-Programming-v2 to strengthen STEM and reasoning. The final SFT stage uses a feedback-driven data curation pipeline that generates errors with an intermediate checkpoint, has a strong judge analyze them, retrieves a weighted 512K-example targeted mixture, and combines it with uniformly-sampled examples for breadth.
  • DPO. In preliminary experiments, AMD found that applying DPO directly to the MoE caused performance regression. Their hypothesis: the load-balancing objective rapidly shifts the experts' router affinity. The fix: disable router bias updates and the auxiliary load-balancing loss during DPO. That's an interesting engineering detail worth flagging.
  • RL. 1,400 GRPO steps on the IF-RLVR subset of Dolci-Think-RL-7B, with asynchronous training and 16K-token responses. AMD layers in zero-gradient signal filtering, active sampling, token-level loss, no KL, clip-higher, no std normalization, and Rollout Routing Replay (R3). Then Multi-Teacher On-Policy Distillation (MOPD) distills the instruction-following expert back into the DPO model.

The scoreboard: matching 4B-scale at a 2.8B-active budget

  • Base model averages 76.7 — strongest among fully open models, ahead of SmolLM3-3B-Base (70.5), OLMo-3-7B (70.1), OLMoE-1B-7B (61.9); WinoGrande 86.5 leads the field; HumanEval+ 65.7; balanced across knowledge, reasoning, math, and code.
  • Think model (SFT → DPO → RL → MOPD) averages 73.22, ahead of Olmo3-7B-Think (71.97), Gemma-4-E4B-think (70.47), and Qwen3.5-4B (69.73). IFEval jumps from 77.08 (DPO) to 83.70; AGIEval 82.50; AIME25 73.40.

In other words, 16B total / 2.8B active lands in the ballpark of leading 4B-scale models. The combination is MoE sparsity + Gated MLA expressivity + AMD-stack communication optimization.

Why this matters: AMD is selling the stack, not the benchmark

A few layers worth unpacking:

First, AMD is not selling a model — it is selling a stack. Instinct MI300X / MI325 GPUs, ROCm software, Primus training framework, Miles RL framework, SGLang inference, and finally the model weights. This is AMD's first complete demonstration of end-to-end LLM training on its own long toolchain — a direct counterpart to NVIDIA's NeMo / Megatron / TensorRT-LLM orbit.

Second, the gap between "fully open" and "open weights only" is real, and AMD fills half of it. Qwen3.8-Max, Llama 4, DeepSeek V4 — the "open weights" camp usually ships only the final SFT or Instruct checkpoint; intermediate states are opaque. Instella-MoE ships all six checkpoints, which means researchers can reproduce stage by stage. That is what "academically reproducible" actually means, and it's why AMD uses the label "state-of-the-art fully open" rather than just "open-source."

Third, Gated MLA and FarSkip-Collective are engineering-level, not architecture-level, innovations. They're not paper-grade breakthroughs. But for AMD's own hardware stack they are load-bearing — AMD's GPU interconnect bandwidth trails NVLink, so FarSkip-Collective's "feed incomplete activations ahead to overlap" trick is essentially algorithm-as-bandwidth-substitute. The 12.7% training speedup and 39.2% TTFT reduction are exactly the hard numbers AMD needs to tell customers "our GPUs run MoE faster, too."

Fourth, this is a sample of "MoE engineering democratization." MoE is no longer a luxury only the frontier labs can afford: 16B total + 2.8B active + 64K context + a complete RLHF pipeline, running on consumer-grade GPU clusters rather than demanding 1,000 H100s. AMD is essentially saying: with a transparent enough stack, mid-scale MoE can complete the RL distillation loop too.

So what

If you build LLMs, the real value of Instella-MoE is the checkpoint ladder + complete training recipe — you can take its Mid-train checkpoint and fine-tune downstream, or reproduce its GRPO modifications. If you're a hardware buyer, FarSkip-Collective and Gated MLA give you empirical evidence that AMD GPUs run MoE without losing to the alternative. If you watch the open ecosystem, this "fully open six-stage" release sets a standard Meta, DeepSeek, and Qwen will be asked to match.

One caveat: AMD ships Instella-MoE under a Research RAIL license — research-only, no commercial use. That limits its short-term commercial reach but actually raises its academic reproducibility value.

References: