[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-kimi-k3-latentmoe-kda-attnres-nope":3,"news-related-3d922c00-afcb-4f1c-a6d5-8f9d6c10c642":41},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","Kimi K3 开源 2.8T MoE 后,Sebastian Raschka 拆解了它的核心架构升级:LatentMoE 压缩 expert 线性层、Kimi Delta Attention + MLA 混合注意力、Attention Residuals 改进残差通路、全场 NoPE 替代 RoPE、原生多模态。这些改动把推理成本只抬高 2%,却压低了验证损失。","# 从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级\n\n2026 年 7 月底,Moonshot AI 把 Kimi K3 权重开源,2.8T 参数 MoE 一举拿下\"目前最大的开源权重模型\"。但比参数规模更值得关注的,是 K3 在架构层面的一系列替换:从 MoE 到 LatentMoE,从标准注意力到多头潜在注意力 + Kimi Delta Attention 的混合方案,从常规残差连接到 Attention Residuals,从 RoPE 全面换成 NoPE。Sebastian Raschka 在发布次日对技术报告做了逐项拆解,这些改动合起来只把推理成本抬高了 2%,却把验证损失和下游性能都再压低了一档。\n\n## 一、LatentMoE:把 MoE 的大线性层也\"潜空间化\"\n\nKimi K3 最显眼的新组件是 LatentMoE。思路和过去一年大模型里\"潜空间化\"的趋势一脉相承——既然多头潜在注意力(MLA)可以通过 down-project 把 attention 矩阵压缩到低秩潜空间,MoE 里的 expert 线性层为什么不能照做?\n\nRaschka 指出,K3 用的 LatentMoE 与 NVIDIA Nemotron 3 Ultra 用的是同一个套路:把 expert 内部的大矩阵先压到一个小维度的潜空间,计算完再升回去。对推理而言,这条路径减少的是 expert 内部的访存和算力,而不是 expert 数量本身。换句话说,它让\"开更多 expert\"这件事变得更便宜,这才是 2.8T 参数仍然能跑得动的核心工程基础。\n\n## 二、Kimi Delta Attention + MLA 的混合:把注意力也压到潜空间\n\nK3 不是单一注意力,而是 hybrid 方案:\n\n- 全局层用 Kimi Delta Attention(KDA),一种循环线性注意力,延续自去年的 Kimi Linear,ETH Zurich 7 月份那篇比较四种循环线性注意力的论文已经把 KDA 列为损失最低的一档。\n- 局部层则走多头潜在注意力(MLA),靠潜空间压缩来减少 KV cache 占用。\n\n两条路一起走的效果是:全局层负责长程依赖但省算力,局部层负责细节但省显存。K3 把这套混合延续到 2.8T 规模,在开源权重里是头一次。\n\n## 三、Attention Residuals:不卷参数卷残差通路\n\n如果说 LatentMoE 和混合注意力都还属于\"用更便宜的方式做同一件事\",那 Attention Residuals(AttnRes)就是这一轮架构改动里唯一的\"非效率性\"升级。\n\nDeepSeek V4 用 mHC(manifold-constrained Hyper-Connections)把残差通路变宽,AttnRes 走的是另一条路:在层与层之间用 attention score 来决定当前残差对下一层的贡献权重。换句话说,残差连接本身带了一个\"哪些层该用多少\"的学习机制。\n\n按技术报告数据,AttnRes 让验证损失和下游任务都\"一致地\"小幅改善,代价是训练成本 +4%、推理成本 +2%。这两个百分比看起来小,但放在 2.8T 规模上,绝对算力是天文数字——Moonshot 愿意为这一点点提升付费,说明残差通路在超大规模上还有红利可挖。\n\n## 四、NoPE 全场:第一个 frontier 级\"无位置编码\"模型\n\nKimi K3 完全抛弃了 RoPE,所有层都用 NoPE(No Positional Embeddings)。Raschka 直接说这是他在 frontier 级模型里见到的第一个全 NoPE 案例——之前要么是 NoPE 用在全局层 + RoPE 用在滑动窗口层,要么是某些小模型试水。\n\nK3 之所以敢这么做,一方面是因为混合注意力里 KDA 已经隐式编码了相对位置信息,另一方面是 MoE 路由本身就提供了位置无关的全局结构。这条路径如果被后续的 V5、DeepSeek V5 跟进,意味着 RoPE 这个从 LLaMA 时代沿用至今的设计可能被加速淘汰。\n\n## 五、原生多模态:架构升级的最后一块拼图\n\nK3 不再是纯文本模型,而是原生多模态。Raschka 把这点单独列出,认为它和上面几个架构改动是同一波技术决策的产物——MoE 让不同模态共享容量,NoPE 让不同模态不需要对齐位置,AttnRes 让跨模态残差可以学习融合权重。一套架构同时解决\"模型够大\"和\"模态够多\"两个问题。\n\n## 个人评论:K3 给\"推理效率战\"立的新标杆\n\n把 K3 这一轮改动和 DeepSeek V4、Nemotron 3 摆在一起,可以看出 2026 下半年开源 frontier 模型的主旋律不再是\"参数更大\",而是\"每一个组件都被换成了推理效率更高的版本\"——MoE 换 LatentMoE,attention 换 MLA + 线性注意力混合,RoPE 换 NoPE。这条路径的意义在于:开源权重模型的可用性不再只看参数,更看推理时每一 token 的真实成本。\n\n对国内开源生态而言,K3 的另一个信号是:2.8T 参数的 MoE 仍然可以做到权重公开 + 推理成本可控,这给 DeepSeek V4 Pro、GLM-5.2 之后的下一轮竞争定了基调——谁能把\"潜空间化\"做到每一层,谁就能在开源 frontier 上保持领先。\n\n最后一个值得关注的点:AttnRes +4% 训练成本换来的\"小幅\"性能提升,在 2.8T 规模上是否值得,完全取决于 Moonshot 后续用 K3 跑多模态 agent 工作流时的实际收益。如果多模态 agent 是 2026 下半年的主战场,那这 4% 可能是今年最划算的一笔架构投资。","https:\u002F\u002Fsebastianraschka.com\u002Fblog\u002F2026\u002Fkimi-k3-architecture-notes.html","8c758013-1efc-4f1d-bc10-8860362115e7",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":25,"name":26,"slug":26,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"858113df-1af3-4f19-aab0-116e78819917","en","From Kimi Linear to Kimi K3: the overlooked MoE upgrade","Sebastian Raschka's architecture teardown of Kimi K3's open 2.8T MoE: LatentMoE compressing expert linear layers, Kimi Delta Attention + MLA hybrid, Attention Residuals rewiring the residual path, full-NoPE across the stack, and native multimodality — all together pushing inference cost up only 2% while validation loss moves down.","# From Kimi Linear to Kimi K3: The Architecture Upgrades Everyone Missed in the MoE Inference-Efficiency Race\n\nWhen Moonshot AI open-sourced Kimi K3 in late July 2026, the headline number was 2.8T parameters — the largest open-weight model in the world. But the parameter count is not the story. The story is what K3 does with every single architecture component: MoE → LatentMoE, vanilla attention → multi-head latent attention + Kimi Delta Attention hybrid, conventional residuals → Attention Residuals, and RoPE → NoPE across the entire stack. Sebastian Raschka's architecture teardown on the day after release walks through each swap. Together they push validation loss and downstream metrics down a notch while only inflating inference cost by 2%.\n\n## 1. LatentMoE: Putting the MoE Linear Layers in a Latent Space Too\n\nThe most visible new component in K3 is LatentMoE. The idea follows the same \"latent-space compression\" trend that has swept large models over the past year: if multi-head latent attention (MLA) can down-project the attention matrix into a low-rank latent space, why can't MoE do the same with the expert linear layers?\n\nRaschka notes that the LatentMoE used in K3 is essentially the same one NVIDIA used in Nemotron 3 Ultra: down-project each expert's large matrix into a smaller latent space, compute there, then up-project back out. For inference, what this saves is memory bandwidth and compute inside each expert — not the expert count itself. In other words, it makes \"running more experts\" cheap enough that a 2.8T-parameter MoE becomes practical to serve.\n\n## 2. Kimi Delta Attention + MLA Hybrid: Attention Also Goes Latent\n\nK3 does not use a single attention mechanism. It runs a hybrid:\n\n- Global layers use Kimi Delta Attention (KDA), a recurrent linear attention inherited from last year's Kimi Linear. ETH Zurich's July paper comparing four cyclic linear attention variants put KDA at the lowest loss.\n- Local layers use MLA, which compresses the KV cache via latent-space projection.\n\nThe two paths do different jobs: global layers handle long-range dependencies cheaply; local layers handle detail cheaply on memory. K3 scales this hybrid to 2.8T parameters — the first time an open-weight frontier model has done so.\n\n## 3. Attention Residuals: Not Cheaper, Just Better\n\nIf LatentMoE and the hybrid attention are both \"cheaper ways to do the same thing,\" then Attention Residuals (AttnRes) is the only non-efficiency upgrade in this round.\n\nDeepSeek V4 widened the residual path with mHC (manifold-constrained Hyper-Connections). AttnRes takes a different route: between layers, an attention score decides how much the current residual should contribute to the next layer. The residual connection itself learns \"how much each layer matters.\"\n\nPer the technical report, AttnRes consistently improves validation loss and downstream metrics, at a cost of +4% training compute and +2% inference compute. Two percent sounds small until you multiply by 2.8T parameters — Moonshot paying that price for a marginal gain tells you there is still headroom in the residual path at frontier scale.\n\n## 4. NoPE Everywhere: The First Frontier-Scale Model Without Positional Embeddings\n\nK3 drops RoPE entirely. Every layer runs NoPE. Raschka calls this the first frontier-level model he has seen go full-NoPE — previous attempts either mixed NoPE in global layers with RoPE in sliding-window layers, or were confined to small models.\n\nK3 can pull this off because the hybrid attention already encodes relative position implicitly through KDA, and MoE routing itself provides a position-independent global structure. If V5, DeepSeek V5, or others follow, the RoPE design we have carried over from the LLaMA era may get sunset faster than expected.\n\n## 5. Native Multimodality: The Last Piece of the Architecture Stack\n\nK3 is no longer text-only. Raschka lists native multimodality alongside the other architectural shifts, and it is in fact a product of the same design decisions: MoE lets different modalities share capacity, NoPE means modalities do not need aligned positional schemes, and AttnRes lets cross-modal residuals learn their own fusion weights. One architecture solves \"big enough\" and \"multi-modal enough\" in the same package.\n\n## Commentary: A New Benchmark in the Inference-Efficiency Race\n\nLooking at K3, DeepSeek V4, and Nemotron 3 side by side, the dominant theme for open-weight frontier models in late 2026 is no longer \"scale up parameters\" — it is \"replace every component with an inference-efficiency-tuned version.\" MoE → LatentMoE, attention → MLA + linear attention hybrid, RoPE → NoPE. What this means in practice: the usability of open-weight frontier models is no longer measured by parameter count alone, but by real per-token inference cost.\n\nFor the Chinese open-source ecosystem specifically, K3 sends a second signal: a 2.8T-parameter MoE can still ship open weights with controllable inference cost. That sets the bar for the next round of competition against DeepSeek V4 Pro and GLM-5.2 — whoever pushes latent-space compression into every layer wins the open-weight frontier.\n\nOne last thing worth watching: whether AttnRes's +4% training cost is justified by downstream multimodal-agent workload performance. If multimodal agents are the main battlefield of late 2026, that 4% may turn out to be the most cost-effective architectural bet of the year.","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00Z","2026-07-30T00:03:33.333667Z","2026-07-30T00:03:33.333676Z",true,"agent",167,{"items":42},[43,48,53,58,63,68],{"id":44,"title":45,"news_slug":46,"published_at":47},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"b5638cab-a4d6-44ac-9230-32ed0a4cba9d","ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文","armt-associative-recurrent-memory-transformer","2026-07-23T00:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"d2844cbb-b70e-469c-95b8-71cee8d735a6","给 Transformer 装上「CNN 鼻子」:用 0.01% 的参数量换 benchmark 普涨","transformer-cnn-nose-0-01-percent","2026-07-22T12:10:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"067a3f68-9bc8-4486-9715-5a391e537909","ETH 统一评测循环线性注意力：Kimi Delta Attention 损失最低","eth-zurich-clvr-kda","2026-07-12T04:00:00+00:00",{"id":69,"title":70,"news_slug":71,"published_at":72},"070aef27-5fdb-4f0a-8b90-99afc1ea34fb","Jet-Long 用「动态双焦 RoPE」让 Qwen3 免训练扩到 128K,RULER 直接多涨 4.79 pp","jet-long-dynamic-dual-rope","2026-07-12T02:30:00+00:00"]