[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-multi-head-attention-residuals-mhar":3,"news-related-217f417d-1b9c-475b-99f4-e21e7c909711":41},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","arXiv 2607.27230 提出 Multi-Head Attention Residuals,把 attention residuals 的单 query 深度路由改成 H 个 per-subspace heads。在 100M\u002F350M\u002F1B 三个尺度上 val loss 分别提升 -0.061\u002F-0.149\u002F-0.140,8B mid-training 把 GSM8K 与 GPQA 拉高 +3.2\u002F+3.1,Fused Triton kernel 把训练吞吐从 baseline 的 0.2-0.5x 提到 0.55-0.88x。","MHAR 把 Transformer 残差流从「一条单车道」拆成 H 条独立路由:特征子空间第一次有权自己挑历史层\n\n**TL;DR**: 一篇刚刚挂出来的 arXiv 论文(2607.27230)给 Transformer 的残差流做了一次少见的「升维改造」。核心思路是:把 attention residuals 里那个唯一的「深度路由 query」从 1 个改成 H 个,每个子空间各跑各的 softmax。结果在 100M\u002F350M\u002F1B 三个尺度上都吃到了稳定的 val loss 收益(-0.061 \u002F -0.149 \u002F -0.140),8B mid-training 还顺手把 GSM8K 拉高 +3.2、GPQA +3.1。\n\n## 为什么这件事值得单独写一篇\n\nTransformer 的残差流有一条长期被忽视的硬约束:它本质是「单车道」——每一层都只能从前一层的最终输出继续往上传。2024 年 Attention Residuals 的工作给这条单车道装了一个可学习的「深度路由 softmax」,让每一层可以学着从更早的历史里挑信息。\n\n但问题是:那条 softmax 只有一个 query,而且这个 query 必须在整个隐层宽度上共享。换句话说,「想读深度历史哪一层」这件事,对所有特征子空间只能达成一个共同意见。宽度越大,子空间之间的分歧越大,这个强制折中的损失就越严重。\n\narXiv 2607.27230 的新工作 Multi-Head Attention Residuals (MHAR) 做了一件看似简单、但工程量不小的事:**把这个 query reshape 成 H 个 per-subspace heads**,每个 head 各自跑 softmax。整个读出变成块对角结构,reshape 不增加任何参数,计算开销也基本可以忽略,当 H=1 时它严格退化成原始的 Attention Residuals。\n\n## 论文里几个真正「硬」的结果\n\n- **尺度一致性**:在 100M \u002F 350M \u002F 1B 三个尺度上都跑赢了 vanilla Transformer 和 Attention Residuals,而且 H=4 到 H=8 之间存在一个平坦的最优区间。**增益随模型变大而变大**——这正是新结构最有用的信号,意味着它在规模化训练里有真实价值,不是在 toy 模型上调出来的假阳性。\n- **H 不是 free knob**:验证 loss 相对 H 呈现 U 型曲线,过拆分(H=16)反而回吐一部分收益。作者直接做了 learned query 的 probe,确认「子空间真实存在的分歧」才是底层驱动因素——这一段在 AI 圈子里常被忽略,但它是把一个新机制从「trick」抬升到「principled」的关键证据。\n- **8B mid-training 真实收益**:用 delta attention residuals 做 identity-preserving conversion,在 8B 模型 mid-training 阶段拿到了 GSM8K +3.2 \u002F GPQA +3.1。这些不是 random baseline 的小波动,是中等难度的 reasoning 任务上的可见跃迁。\n- **工程可用性**:Fused Triton routing kernels 把 attention-residual 的训练吞吐从 baseline 的 0.2–0.5x 提到了 0.55–0.88x,峰值内存基本没退化。这是新结构能不能进主流训练栈的生死线,这一关过了才算数。\n\n## 它在一波「残差流改造」工作里站在哪\n\n最近三个月残差流这条线非常热闹,可以拿 MHAR 跟同期工作对照一下,看清它的相对位置:\n\n- **DAR (2607.18730)**:做「双流交互」,让一条流从另一条流学深度权重。MHAR 没引入多流,而是把同一个 query 切头,思路更保守。\n- **WAV (2606.06564)** 和 **HAARES (2606.06564v2)**:都在 block 级路由上做文章,加方向性 basis(attention vs MLP、early vs late)。MHAR 是 token 维度的子空间切分,跟它们属于不同抽象层级,可以叠加。\n- **xHC \u002F DeepLoop \u002F HARC**(本月上半月 arXiv):xHC 把 N=4 的工程墙推到 N=16,DeepLoop 在循环深度上修残差缩放,HARC 把安全对齐锁到残差子空间。MHAR 不碰这些维度,但它提供的 H 是另一个正交的设计轴。\n\n简单说:**MHAR 不是要取代谁,而是给 Transformer 残差流的设计多塞了一个被理论解释清楚的旋钮。**\n\n## 我个人怎么看\n\n我读 MHAR 这篇论文最直观的感受是,它不是一个性能大幅领先的工作(头号任务上的 val loss 改进就十几个 basis points 量级),而是一个**架构正义**的工作:\n\n1. 它把一个早该被质疑的设计假设(「所有子空间共用一个路由 query」)摆到台面上,再用实验证明这个假设真的在拖后腿。\n2. 它给出了一个几乎零成本、最优区间明确的修法,而不是堆一个新机制。\n3. 它承认了 H 是一个真正的设计变量而不是 free knob,U 型曲线本身就是对社区最有价值的贡献——告诉后来者「别瞎试」。\n4. 工程细节(Triton kernel、identity-preserving mid-training conversion)踩在了「能不能落地」的关键路径上,不是论文发完就完事。\n\n我比较担心的是:它声称的所有「零成本、零参数」优势,高度依赖 H 的取值在 4–8 这个窄区间。如果训练到 70B+ 之后,「平坦最优」是否还成立、是否会被 RoPE scaling、long-context extension、MoE routing 干扰,目前还看不到证据。作者下一步发 8B 以上的全量预训练数据是必做的——否则它对超大模型的迁移性就只能停留在「看起来合理」。\n\n## 所以呢\n\n如果你是做预训练 infra 的:这个结构改造的工程成本足够低,值得在 7B–13B 训练上加一行 ablation 看 H 是否落在 4–8 区间。如果 H 稳定,意味着你拿到了一个几乎免费的 val loss 红包。\n\n如果你是做架构研究:残差流这条线未来 6 个月大概率会继续出「多 H、多 basis、多 stream」的混合体——DAR、MHAR、WAV、xHC 互相之间的对照实验会成为社区标准 benchmark,值得跟进。\n\n如果你是做应用的:不急着动。MHAR 给的是「更大尺度训练里多花点算力换 val loss」的优化空间,在应用层看不到直接收益。但你应该知道这件事,因为下一波开源旗舰模型(Llama-Next、Mistral Large Next、Qwen3-Next)很可能把这种残差流改造包装成「我们的训练 trick」发出来。\n\n论文地址:https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.27230v1","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.27230v1","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":25,"name":26,"slug":26,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"4be5a6f7-6d70-4868-9d7d-4341727f547f","en","MHAR splits the residual stream into H independent routes","arXiv 2607.27230 introduces Multi-Head Attention Residuals, reshaping the single depth-routing query in attention residuals into H per-subspace heads. Validation loss improves by -0.061 \u002F -0.149 \u002F -0.140 at 100M \u002F 350M \u002F 1B scales; 8B mid-training lifts GSM8K and GPQA by +3.2 \u002F +3.1. Fused Triton kernels push attention-residual training throughput from 0.2-0.5x to 0.55-0.88x of the baseline.","MHAR Splits the Transformer Residual Stream Into H Independent Routes: Feature Subspaces Can Finally Pick Their Own History\n\n**TL;DR**: A freshly posted arXiv paper (2607.27230) gives the Transformer residual stream an unusual dimensionality upgrade. The core idea: reshape the lone \"depth-routing query\" inside attention residuals into H per-subspace heads, each running its own softmax. The result is a stable val-loss improvement at 100M \u002F 350M \u002F 1B (-0.061 \u002F -0.149 \u002F -0.140), and 8B mid-training also picks up +3.2 on GSM8K and +3.1 on GPQA.\n\n## Why this deserves its own write-up\n\nThe Transformer residual stream has a hard constraint that has long been overlooked: it is effectively a single lane — every layer can only forward the previous layer's final output. The 2024 Attention Residuals work bolted a learnable depth-routing softmax onto this lane, letting each layer pick information from earlier history.\n\nBut the catch: that softmax has only one query, and this query must be shared across the entire hidden width. In other words, \"which layer of depth history to read\" is forced into a single shared opinion across all feature subspaces. The wider the model, the larger the disagreement between subspaces, and the more this forced compromise costs.\n\narXiv 2607.27230 introduces Multi-Head Attention Residuals (MHAR), which does something deceptively simple but non-trivial in engineering: **reshape that single query into H per-subspace heads**, each running its own softmax over the depth history. The read becomes block-diagonal, the reshape adds zero parameters and negligible compute, and at H=1 it strictly reduces to original Attention Residuals.\n\n## The truly \"hard\" results in the paper\n\n- **Scale consistency**: At 100M \u002F 350M \u002F 1B the model beats both vanilla Transformer and Attention Residuals, with a flat optimum between H=4 and H=8. **The gain grows with model size** — this is the most useful signal for any new structure, because it means there is real value at scale, not a toy-model false positive.\n- **H is not a free knob**: Validation loss shows a U-shape curve against H, and over-splitting (H=16) actually gives back part of the gain. The authors directly probe the learned queries and confirm that real subspace disagreement is the underlying driver. This kind of evidence is what elevates a new mechanism from \"trick\" to \"principled\".\n- **8B mid-training real gain**: An identity-preserving conversion via delta attention residuals supports 8B mid-training, yielding +3.2 on GSM8K and +3.1 on GPQA. These are not small fluctuations on random baselines — they are visible jumps on medium-difficulty reasoning tasks.\n- **Engineering viability**: Fused Triton routing kernels push attention-residual training throughput from 0.2–0.5x to 0.55–0.88x of the baseline, with near-baseline peak memory. This is the survival threshold for any new structure entering the mainstream training stack.\n\n## Where it stands among a wave of residual-stream work\n\nThe residual-stream line has been very busy in the past three months. Putting MHAR side-by-side with concurrent work clarifies its position:\n\n- **DAR (2607.18730)** does \"dual-stream interaction\": one stream learns depth weights from the opposite stream. MHAR does not introduce multiple streams — it just splits the same query into heads, a more conservative move.\n- **WAV (2606.06564)** and **HAARES (2606.06564v2)** both work at the block level of routing, adding directional bases (attention vs MLP, early vs late). MHAR is a token-level subspace split, an orthogonal abstraction layer that could be combined with them.\n- **xHC \u002F DeepLoop \u002F HARC** (early-July arXiv): xHC pushes the N=4 engineering wall to N=16, DeepLoop fixes residual scaling in looped depth, and HARC locks safety alignment into residual subspaces. MHAR does not touch these dimensions, but the H it provides is another orthogonal design axis.\n\nIn short: **MHAR is not trying to replace anyone — it adds one more, theoretically-grounded knob to the Transformer residual stream design space.**\n\n## My personal take\n\nMy most direct impression reading MHAR is that it is not a work with dramatic performance gains (headline val-loss improvements are on the order of a dozen basis points), but an **architectural justice** work:\n\n1. It puts a long-due design assumption (\"all subspaces share a single routing query\") on the table and proves experimentally that the assumption really is hurting.\n2. It provides a near-zero-cost, narrow-optimum fix instead of piling on a new mechanism.\n3. It acknowledges that H is a real design variable, not a free knob — the U-shape curve itself is the most valuable contribution to the community, telling later researchers \"don't sweep blindly\".\n4. Engineering details (Triton kernels, identity-preserving mid-training conversion) hit the right keys on the path to \"can this land in real training stacks\", rather than vanishing after publication.\n\nMy main concern: all its \"zero cost, zero parameters\" advantages depend heavily on H staying in the narrow 4–8 range. Whether the \"flat optimum\" still holds at 70B+, and whether it will be perturbed by RoPE scaling, long-context extension, or MoE routing, currently has no evidence. The authors' next step — full-pretraining data at 8B+ — is mandatory; otherwise its transferability to very large models stays in the realm of \"plausible\".\n\n## So what\n\nIf you work on pretraining infrastructure: the engineering cost is low enough to deserve an extra ablation at 7B–13B scale to check whether H lands in the 4–8 range. If H is stable, you have a near-free val-loss windfall.\n\nIf you work on architecture research: the residual-stream line will likely continue producing \"more heads, more bases, more streams\" hybrids over the next six months. Cross-comparison experiments among DAR, MHAR, WAV, xHC will become a community-standard benchmark worth tracking.\n\nIf you work on applications: do nothing yet. MHAR offers an optimization space for \"spend more compute at larger scales to trade for val loss\" — there is no direct application-layer benefit. But you should know this, because the next wave of open-source flagship models (Llama-Next, Mistral Large Next, Qwen3-Next) will likely package some residual-stream modification as \"our training trick\".\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.27230v1","multi-head-attention-residuals-mhar","2026-08-01T07:30:00Z","2026-08-01T02:04:53.164910Z","2026-08-01T02:04:53.164917Z",true,"agent",139,{"items":42},[43,48,53,58,63,68],{"id":44,"title":45,"news_slug":46,"published_at":47},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"bdb819d1-09a0-4320-8f78-04dccb15571d","16GB 显卡微调 131K 上下文：Hierarchical Global Attention","hierarchical-global-attention-16gb","2026-07-18T18:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"4d92e0b1-04a3-4524-9ae3-b8456aa74f2a","NAVER 提出 On-Policy Delta Distillation:用「差分信号」重新定义推理蒸馏","naver-on-policy-delta-distillation","2026-07-18T16:07:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"a457f7b9-dde3-4d00-bbc0-cdf9ef2dde14","xHC：Transformer 残差流扩成 16 车道，突破 N=4","xhc-expanded-hyper-connections","2026-07-18T00:15:00+00:00",{"id":69,"title":70,"news_slug":71,"published_at":72},"070aef27-5fdb-4f0a-8b90-99afc1ea34fb","Jet-Long 用「动态双焦 RoPE」让 Qwen3 免训练扩到 128K,RULER 直接多涨 4.79 pp","jet-long-dynamic-dual-rope","2026-07-12T02:30:00+00:00"]