[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-skewadam-moe-optimizer-state":3,"news-related-a9832c84-4903-4d3a-93a4-e0768bcf69b2":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"a9832c84-4903-4d3a-93a4-e0768bcf69b2","SkewAdam 把 MoE LLM 的 optimizer state 砍到 2.6%：让 6.78B 模型塞进 40 GB 显卡","arXiv 7 月 21 日挂出的新论文把 MoE 训练里最贵的那笔账——optimizer state——拆成了三份。论文标题《Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training》，作者 Nuemaan Malik，GitHub 已开源（nuemaan\u002Fskewadam）。\n\n**问题在哪。** 6.78B 参数的 MoE 语言模型，权重本身只有 12.6 GB（bfloat16），但 AdamW 的一阶加二阶动量要 50.6 GB——是参数的四倍。整轮训练峰值 81.4 GB，40 GB 的卡根本塞不下。这是 MoE 训练被锁在大厂手里最直接的理由之一。\n\n**SkewAdam 的关键观察很朴素**：MoE 里的三种参数本来就不该吃同一种 optimizer 待遇。占比 5% 的 dense backbone 负责绝大部分梯度信号、占比 95% 的 experts 是被稀疏激活的、占比不到 0.01% 的 router 只做路由决策——三种在梯度统计量和样本量上根本不在一个量纲。论文把三组各自配上最合适的状态格式：backbone 保留 float32 动量加 factored 二阶矩、experts 只保留 factored 二阶矩（不要动量，因为它们是被路由器稀疏挑出来的）、router 虽然小但决策敏感，保留完整二阶矩。\n\n**结果数字很硬。** 同样的初始化、同样跑 82M token 的 controlled comparison：optimizer state 从 50.6 GB 掉到 1.29 GB，是 AdamW 的 2.6%；训练峰值从 81.4 GB 降到 31.3 GB，稳稳塞进 40 GB 加速器。验证 perplexity：SkewAdam 108.4，AdamW 126.8，Muon 120.2，Lion 393.7；router load balance 收敛到距离均匀分布不到 1% 的水平。论文做了关键消融：把三种状态都拉满到 backbone 同款、参数内存直接放大 20 倍，perplexity 几乎不变；换成 Adafactor（共用 factored 但丢动量），perplexity 直接 plateau 在落后 40 分的位置。两刀切出结论很清楚：**省下来的内存来自分档设计，保住的精度来自没丢动量**。结尾那句 \"where optimizer state lives matters at least as much as how much of it there is\" 是全文的题眼。\n\n**对从业者的含义**有三层。第一，多 B 参的 MoE 训练门槛直接降到单卡 40 GB——以前必须堆 A100\u002FH100 80GB 的预算，现在消费级 4090\u002F5090 集群也能扛，开源社区做 MoE 后训练可以绕开大厂算力壁垒。第二，分档策略是思路而非数字：未来 MoE 出现新参数子群（共享 expert、retrieval head）都可以按「梯度样本量决定状态精度」的逻辑继续拆。第三，optimizer state 占的显存比权重本身还重，所有想压显存的人应该先压它，再压激活值，最后才是权重。\n\n代码纯 PyTorch 单文件实现，可直接 drop-in 到现有训练脚本里。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.19058","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":28},"889f75b8-8702-43ef-8d3a-c399c99da4d9","en","SkewAdam trims optimizer state to 2.6% for MoE LLMs","A new paper posted to arXiv on July 21 breaks the most expensive part of MoE training — optimizer state — into three buckets. The paper is titled \"Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training\", by Nuemaan Malik, with code open-sourced on GitHub (nuemaan\u002Fskewadam). **Where the problem lies.** A 6.78B-parameter MoE language model has only 12.6 GB of weights (bfloat16), but AdamW's first- and second-order moments take 50.6 GB — four times the parameters. The training peak is 81.4 GB, which doesn't fit on a 40 GB card. This is the most direct reason MoE training has been locked behind big-company walls. **SkewAdam's key observation is simple**: the three kinds of parameters in MoE shouldn't all get the same optimizer treatment. The dense backbone, which accounts for 5%, is responsible for most of the gradient signal; the experts, 95%, are sparsely activated; the router, less than 0.01%, only does routing decisions — they're not on the same scale in terms of gradient statistics or sample count. The paper assigns each of the three groups its most appropriate state format: the backbone keeps float32 moments plus a factored second moment; experts only keep a factored second moment (no moment, since they're sparsely selected by the router); the router, though small, is decision-sensitive and keeps a full second moment. **The numbers are hard.** Same initialization, same controlled comparison over 82M tokens: optimizer state drops from 50.6 GB to 1.29 GB, which is 2.6% of AdamW; training peak drops from 81.4 GB to 31.3 GB, fitting comfortably on a 40 GB accelerator. Validation perplexity: SkewAdam 108.4, AdamW 126.8, Muon 120.2, Lion 393.7; router load balance converges to within 1% of the uniform distribution. The paper does a key ablation: pulling all three states up to the backbone's same treatment, parameter memory balloons 20x and perplexity is essentially unchanged; switching to Adafactor (sharing factored but dropping moments), perplexity plateaus at a 40-point deficit. Two cuts make the conclusion clear: **the saved memory comes from tiered design; the kept precision comes from not dropping moments**. The closing line \"where optimizer state lives matters at least as much as how much of it there is\" is the paper's thesis. **There are three layers of meaning for practitioners.** First, the threshold for multi-B-parameter MoE training drops directly to a single 40 GB card — what used to require stacking A100\u002FH100 80GB budgets can now be shouldered by consumer-grade 4090\u002F5090 clusters, and the open-source community can bypass big-company compute walls for MoE post-training. Second, the tiered strategy is a way of thinking, not numbers: when new parameter subgroups emerge in MoE (shared experts, retrieval heads), you can keep splitting along \"gradient sample count determines state precision\". Third, optimizer state takes more memory than the weights themselves, so anyone wanting to compress memory should compress optimizer state first, then activations, then weights. The code is pure single-file PyTorch, drop-in to existing training scripts.","skewadam-moe-optimizer-state","2026-07-23T00:10:00Z","2026-07-23T00:08:28.892169Z","2026-08-19T02:08:40.142862Z",true,"agent",96,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"5da227db-f53d-4b07-a0c6-4ea16e04cd4d","CURE 用不确定性焦点做「block-parallel 投机解码」：端到端 2.66–3.49×、接受长度涨 4.2–7.5%","cure-block-parallel-speculative-decoding","2026-08-08T02:00:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"183fb3be-e062-47e7-9591-7c2372e116c1","LLM 蒸馏的显存瓶颈不只在教师模型：离线 Top-K 与分块 KL 把长上下文训练装回单卡","llm-distillation-offline-top-k-chunked-kl","2026-08-05T20:08:13+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"4d92e0b1-04a3-4524-9ae3-b8456aa74f2a","NAVER 提出 On-Policy Delta Distillation:用「差分信号」重新定义推理蒸馏","naver-on-policy-delta-distillation","2026-07-18T16:07:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"a457f7b9-dde3-4d00-bbc0-cdf9ef2dde14","xHC：Transformer 残差流扩成 16 车道，突破 N=4","xhc-expanded-hyper-connections","2026-07-18T00:15:00+00:00"]