[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-google-muir-linear-attention":3,"topics-all":38,"news-related-2d4e00e8-93d2-4a76-ad68-a362493ea3a0":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"2d4e00e8-93d2-4a76-ad68-a362493ea3a0","Google线性注意力Muir:4K训练外推千万token","Google 团队在 arXiv 发布线性注意力 Muir：证明擦除门过度参数化，把门控降秩、匹配滤波读出重配状态行，省下参数回填更宽 FFN。166M-1.34B 困惑度低于 GDN-2 与 Transformer；4K 训练外推 10M token 不漂移；32K 训练比 Transformer 快 2.7 倍。","线性注意力想解决的问题很朴素：Transformer 的 softmax 注意力，训练算力随序列长度平方增长，推理缓存随长度线性膨胀。把上下文压进一个固定大小的 key-value 状态，训练变线性、推理内存恒定——这是 Gated DeltaNet、Kimi Linear、GDN-2 这条门控 delta 规则族谱的共同卖点。11 月 5 日，Google 研究者（Tushaar Gangavarapu 等）在 arXiv 放出新作 Muir，做的事本质上是一次「参数预算审计」：先证明这套架构里的擦除门被过度参数化，再把省下来的预算花回该花的地方。\n\n## 两个被忽视的观察\n\n论文给出两条分析。其一，状态更新对「擦除」的敏感度远低于对「写入」的敏感度——key 激活和单位归一化把 key 能量集中到稀疏的通道子集上，而 value 写入会缩放所有通道，所以擦除门完全可以用更低的秩来参数化。其二，慢衰减状态行持续累积 value 写入，范数比快行大一个数量级以上；在无权重的读出里，这些高范数慢行占主导，逼着模型在写入时加大擦除力度来压制干扰，结果是已存好的关联被提前覆写。\n\n## Muir 怎么花这笔预算\n\n对此 Muir 做三步重分配：把全部递归与输出门控融合成按敏感度分配的低秩投影；用匹配滤波读出门（静态通道轮廓叠加逐 token 动态调制）重新加权状态行；把省下的参数回填 FFN。在 GDN-2 里，两个全秩 d×d 门控矩阵要吃掉 2d²，占每层 12d² 预算的六分之一，等参数约束下 FFN 被挤窄 25% 以上。Muir 把 FFN 宽度拉宽 30.4%（5,888 到 7,680），编译后每步 FLOPs 反而降 1.8%，HBM 流量降 0.7%-1.4%。FFN 正是事实性关联的仓库，这笔回填直接换成记忆容量。\n\n## 数字说话\n\n166M-1.34B 全档测试：困惑度比 GDN-2 低 1.8%-2.0%，比 Transformer 低 2.5%-3.6%，常识准确率最高提升 1.9 分。1.34B 档上多跳与多针检索超过两个基线；从 4K 训练长度外推 2,560 倍到 10M token 流，困惑度不漂移；静态读出下 1M token passkey 检索保持近 100%，而 GDN-2 过了 4K-8K 就崩。工程侧同样硬：吞吐与 GDN-2 持平，100B token 验证损失提前 1.2 倍达到，32K 长度下比 Transformer 训练快 2.7 倍。这背后的 kernel 是为宽矩阵单元重新设计的 boundary-pivoted chunkwise 更新——在 88 块 TPU7x Ironwood 上，256×256 脉动阵列的矩阵单元吞吐约是向量单元的 64 倍，新 kernel 把 98% 的 FLOPs 落在矩阵单元上，绕开标准 chunkwise 算法在大 chunk 下的数值上下溢。\n\n## 冷静看\n\n论文也交代了边界：短上下文 Transformer 仍然更快（T≤8K），Splash Attention 在 11K 以下占优，纯 JAX 实现的 XLA 边界 HBM 读写是主要开销；静态读出在多跳任务上跟不上动态读出，作者建议用滑动窗口注意力做混合。另外，权重和代码还没放（HF 页关联模型数为 0），1.34B 也远非前沿规模。\n\n所以呢：线性注意力的竞争正在从「换一种架构」进入「抠每一份预算」的阶段——在固定参数量下，谁把门控花在刀刃上，谁就能在长上下文外推上白捡数量级。族谱上 Kimi Linear、GDN-2、Muir 一路迭代，说明这条路远没收敛。论文原文见 [arxiv.org\u002Fabs\u002F2611.08376](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08376)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08376","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"8cf7490f-2449-4ba7-be19-61befa0d92b4","google",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"9855ba39-f93a-4e4d-a544-f59584edda3f","en","Google's Muir Linear Attention: 4K Training, 10M Tokens","Google's Muir demotes over-parameterized erase gates to low rank and reinvests the savings in a wider FFN, extrapolating 4K training to 10M-token streams.","Linear attention attacks a simple problem: softmax attention scales quadratically with sequence length during training and keeps a linearly growing cache at inference. Compressing context into a fixed-size key-value state buys linear-time training and constant-memory inference — the shared pitch of the gated delta-rule family, from Gated DeltaNet through Kimi Linear to GDN-2, which generalizes the line with channel-wise decay and decoupled full-rank erase\u002Fwrite gates. On November 5, three Google researchers — Tushaar Gangavarapu, Gil Tabak, and Clemens JS Schaefer — posted Muir to arXiv. The work is essentially a parameter-budget audit: first prove that the erase gates in this architecture are over-parameterized, then reinvest the savings where they actually pay.\n\n## Two overlooked observations\n\nThe paper leads with analysis rather than architecture. First, state updates are far less sensitive to erasure than to writes: key activation and unit normalization concentrate key energy onto a sparse subset of channels, while value writes scale all channels — so the erase gate can safely be parameterized at lower rank. Second, slow-decaying state rows keep accumulating value writes over hundreds to thousands of tokens, reaching norms more than an order of magnitude larger than fast rows. Under unweighted readout these high-norm slow rows dominate the output, forcing stronger write-time erasure to suppress interference — which prematurely overwrites the very associations the state was supposed to keep.\n\n## How Muir spends the budget\n\nGuided by that analysis, Muir makes three reallocations. It fuses all recurrence and output gates into a sensitivity-allocated low-rank projection. It re-weights state rows with a matched-filter readout gate that combines a static channel profile with dynamic per-token modulation. And it reinvests the saved parameters into a wider feed-forward network. The accounting is concrete: in GDN-2, two dense d×d gate projections consume 2d² parameters — one-sixth of the 12d² per-layer budget — squeezing FFN width by over 25% under iso-parameter constraints. Muir widens the FFN by 30.4% (from 5,888 to 7,680) while cutting compiled per-step FLOPs by 1.8% and HBM traffic by 0.7–1.4%. Since the FFN is where factual associations live, the reinvestment converts directly into memory capacity.\n\n## The numbers\n\nAcross 166M–1.34B parameters, Muir lowers perplexity by 1.8–2.0% over GDN-2 and 2.5–3.6% over Transformer, and raises commonsense accuracy by up to 1.9 points. At 1.34B it beats both baselines on multi-hop and multi-needle recall, extrapolates 2,560× from its 4K training length to 10M-token streams without perplexity drift, and keeps near-100% passkey retrieval at 1M tokens with static readout — whereas GDN-2 collapses past 4K–8K. The engineering side is just as sharp: Muir matches GDN-2 throughput, reaches its 100B-token validation loss 1.2× faster, and trains 2.7× faster than Transformer at 32K.\n\nThe kernel behind this is a boundary-pivoted chunkwise update designed for wide matrix units. On the 88 TPU7x Ironwood chips used in the paper, each chip pairs two TensorCores — 256×256 systolic arrays delivering 2,307 TFLOP\u002Fs in bfloat16 — with a vector unit roughly 64× slower. Standard chunkwise algorithms divide keys by cumulative decay prefixes, which underflows on fast-decaying channels at the large chunk sizes these accelerators need. Muir's kernel splits relative decays without division and solves within-chunk triangular systems via block matrix multiplications, keeping 98% of FLOPs on the matrix units.\n\n## Keep it cool\n\nThe paper is candid about limits. Transformer remains faster at short contexts (up to 8K), and Splash Attention wins below roughly 11K, partly because the pure-JAX implementation pays XLA kernel-boundary HBM traffic between chunk stages. Static readout cannot match varying lags on multi-hop and extractive tasks — dynamic readout is what holds perplexity flat across 10M-token streams — and the authors propose pairing sliding-window attention with static-readout Muir as a hybrid. No weights or code have been released: the Hugging Face paper page lists zero linked models, and 1.34B is far from frontier scale.\n\nThe takeaway: linear-attention competition is shifting from \"replace the architecture\" to \"audit every parameter.\" Under a fixed budget, whoever spends gate capacity where it matters gets orders-of-magnitude long-context extrapolation nearly for free. With Kimi Linear, GDN-2, and Muir iterating along one family tree, this path is far from converged. The paper is at [arxiv.org\u002Fabs\u002F2611.08376](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08376).","google-muir-linear-attention","2026-11-07T17:10:00Z","2026-11-07T17:09:13.473539Z","2026-11-07T17:09:13.473549Z",true,"agent",254,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"abd8eb9f-c106-4899-bf24-b6bbed3da5e3","EPFL论文实测:注意力砍维度而非砍token,只算三成点积","hisa-hidden-dimension-sparse-attention-rope","2026-11-20T09:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"f3edd5fd-5a5c-447f-8704-500856b96d09","ZHA:Zyphra 用树形索引把注意力成本压成次二次","zyphra-zha-tree-indexed-subquadratic-attention","2026-11-02T05:21:18+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"2f3e98d3-1168-470f-8cf3-6b0d09ee0ada","1280亿token实测:cosine调度输给1\u002F√t衰减","air-elr-schedule-weight-normalized","2026-10-31T21:12:47+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"49972f78-da29-4041-8f1f-c57d9d8ceeac","MIT 一行代码修 MTP:八成梯度噪声被滤掉","soft-mtp-kl-gradient-recovery","2026-10-30T17:12:42+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"a840ee24-8865-4038-986e-1398549feeb0","苹果论文重算循环大模型算力账:停止头说省一半,实测跑 7.6 轮","apple-halting-head-looped-lm-compute-audit","2026-10-26T03:05:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"fd29c2f1-175f-416e-bde2-95dd65ca3465","KronLSTM 精准改写记忆:线性注意力长上下文新解","kronlstm-kronecker-delta-rule","2026-10-20T15:06:16+00:00"]