[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-eth-zurich-clvr-kda":3,"news-related-067a3f68-9bc8-4486-9715-5a391e537909":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"067a3f68-9bc8-4486-9715-5a391e537909","ETH 统一评测循环线性注意力：Kimi Delta Attention 损失最低","让 LLM 处理长上下文最大的瓶颈是 softmax 注意力的平方复杂度。过去一年,基于 delta-rule 的循环线性注意力在 Mamba \u002F RWKV 之后爆发出 DeltaNet、Gated DeltaNet、Kimi Delta Attention、Gated DeltaNet-2 等多个变体——但它们各自的读写控制记忆衰减差异几乎不透明,工程团队无法判断该选哪种。\n\n7 月 8 日,ETH Zurich 的 Cerruti 等人在 arXiv:2607.07953 把四种循环线性注意力用统一的循环记忆符号重新表达,并在 350M 参数、15B tokens 的匹配设定下做了横向对比。三条关键观察:**一、Kimi Delta Attention + Muon 优化器在最终验证损失上最低**,反映 Moonshot 在长上下文架构上的领先;**二、纯 Gated DeltaNet + AdamW 训练吞吐最高**,而 hybrid(混合 softmax 与循环层)虽然在损失上略胜,但吞吐显著下降;**三、Muon 优化器在所有匹配架构下都稳定优于 AdamW**,提示 community 应重新审视优化器选择。\n\n团队还提出一个轻量跨层路由机制 CLVR (Cross-Layer Value Routing):把上一层的写值路由到下一层的 hidden stream,而不是更直觉的写误差。在 DeltaNet 与 Gated DeltaNet 上,CLVR 都让最终验证损失下降。\n\n这篇论文的真正价值不是某项突破,而是把过去一年碎片化的 linear attention 实验拉到同一张控制变量表上——但它没有测推理延迟,而真正上线最在意的恰恰是 1k–100k 上下文下的每 token 推理速度。配套代码已开源,值得复现到下游长上下文模型。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.07953","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"a13c5df1-91eb-4dea-8506-15dd6f4143c5","en","ETH benchmarks recurrent linear attention: Kimi Delta wins","The biggest bottleneck on letting LLMs handle long contexts is the quadratic complexity of softmax attention. In the past year, delta-rule-based recurrent linear attention has exploded with multiple variants after Mamba \u002F RWKV — DeltaNet, Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2 — but the read\u002Fwrite control memory decay differences between them are nearly opaque, and engineering teams can't decide which to pick. On July 8, ETH Zurich's Cerruti et al. re-expressed four recurrent linear attentions using a unified recurrent-memory notation in arXiv:2607.07953, and ran a horizontal comparison at matched settings of 350M parameters and 15B tokens. Three key observations: **one, Kimi Delta Attention + the Muon optimizer has the lowest final validation loss**, reflecting Moonshot's lead in long-context architectures; **two, pure Gated DeltaNet + AdamW has the highest training throughput**, while hybrid (mixing softmax and recurrent layers) wins slightly on loss but throughput drops significantly; **three, the Muon optimizer stably beats AdamW across all matched architectures**, suggesting the community should revisit optimizer choice. The team also proposes a lightweight cross-layer routing mechanism, CLVR (Cross-Layer Value Routing): route the previous layer's write value to the next layer's hidden stream, rather than the more intuitive write error. On both DeltaNet and Gated DeltaNet, CLVR reduces final validation loss. The real value of this paper isn't any single breakthrough, but pulling the past year's fragmented linear attention experiments onto the same controlled-variable table — but it doesn't measure inference latency, while what really matters for production is per-token inference speed at 1k–100k context. The companion code is open-sourced and worth reproducing into downstream long-context models.","eth-zurich-clvr-kda","2026-07-12T04:00:00Z","2026-07-12T04:06:03.934019Z","2026-08-19T02:08:40.142862Z",true,"agent",107,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"b5638cab-a4d6-44ac-9230-32ed0a4cba9d","ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文","armt-associative-recurrent-memory-transformer","2026-07-23T00:00:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"d2844cbb-b70e-469c-95b8-71cee8d735a6","给 Transformer 装上「CNN 鼻子」:用 0.01% 的参数量换 benchmark 普涨","transformer-cnn-nose-0-01-percent","2026-07-22T12:10:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"29774f38-c361-4dca-b11a-c14df2fc84d9","HiLS 把\"无限上下文\"从口号变成数学:让稀疏注意力首次跑赢 Full Attention","hils-hierarchical-landmark-sparse","2026-07-07T14:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"5c53c383-9727-4880-95e9-9fa752132b01","把混合注意力推到 head 级：HydraHead 用 7:1 LA\u002FFA 比实现 3:1 层混的长上下文性能","hydrahead-7-to-1-la-fa-head-mixed-attention","2026-06-20T16:14:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00"]