Although MoE architectures are celebrated for sparse activation, all experts still need to reside in VRAM — and that keeps deployment costs stubbornly high. arXiv 2606.27866 introduces FlexMoE, a "train-once-and-get-the-whole-budget-stack" approach: each expert's FFN channels are sorted by importance, every expert learns a discrete action that prunes its low-weight channels, and progressive compression then exports a nested sub-network family from high to low budgets out of a single training run. In other words, one run delivers a "budget-elastically stretchable" nested family of sub-networks.
More impressive is its "cross-budget transfer" design: a single recovery fine-tune at the medium 40% budget lets the recovered model transfer directly to unseen budget tiers, with no retraining. On Qwen2-57B-A14B the paper shows striking fidelity — without fine-tuning, dropping 50% of routed expert parameters still preserves 99.8% of base performance; pushing further down yields real VRAM savings and throughput gains at deployment, with runtime online budget switching that removes the need to compress a separate weight for each SLA.
FlexMoE puts "nested structure + multi-tier output from one training run" squarely in front of MoE LLMs: inference providers need only one FlexMoE-style checkpoint to switch seamlessly between low-resource edge and high-throughput data-center environments. The authors place "kernel-level co-design" and "online budget switching" at the end on purpose — to tell the industry that MoE deployment now has "elastic budget scaling" as an engineering capability, a pragmatic tool any MoE model from Mistral / DeepSeek / Qwen can adopt directly.