[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-hydrahead-7-to-1-la-fa-head-mixed-attention":3,"news-related-5c53c383-9727-4880-95e9-9fa752132b01":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"5c53c383-9727-4880-95e9-9fa752132b01","把混合注意力推到 head 级：HydraHead 用 7:1 LA\u002FFA 比实现 3:1 层混的长上下文性能","长上下文这条赛道上，「混合注意力」已经基本是标准动作——把若干层 Full Attention（FA）保留、其余换成 Linear Attention（LA）以压制二次复杂度。但不管是 Qwen 3.5、DeepSeek V4 还是 NVIDIA 的 Gated DeltaNet 系列，混合都是 layer-wise 的，按层一刀切。\n\nZhentao Tan 等人的 HydraHead（arXiv 2606.20097）做了一件听起来朴素但工程意义很大的事：把混合粒度从 layer 推到 head。作者先做了一层 interpretability 分析，发现同一层内的不同 attention head 在功能上有显著差异——有的 head 负责精确检索，有的偏向局部平滑。基于这一观察，他们提出两个关键设计：\n\n1. interpretability-driven selection：通过分析 head 的检索贡献，把 FA 只留给那些 retrieval-critical head，其余 head 走 LA；\n2. scale-normalized fusion module：用一个 scale-normalized 的融合模块，协调 FA 与 LA 在输出分布上的差距，避免两种注意力拼接后被一方的范数压住。\n\n更关键的是它带来的算力账：用 7:1 的 LA\u002FFA 比例，就能匹配 3:1 layer-wise hybrid 的长上下文能力——也就是说，每保留一个 FA head，能省下 4 个原本必须保留 FA 的 layer 的开销。在 512K 上下文下，HydraHead 只训练 15B tokens 就相对基线拿到 69% 的提升，逼近 Qwen3.5（原生 256K 上下文、同尺寸 SOTA）的水平。\n\n对工业界来说，这条路最有意思的不是再多一个 benchmark 第一，而是它打开了一个用 interpretability 驱动架构设计的范例：先看清 head 的功能分工，再决定哪些 head 走哪条路径。这与 Gated DeltaNet-2 在注意力算子层面「擦写分离」、HeraBench 用故障注入评测长程 agent，是同一波「长上下文从堆参数转向细粒度结构」的工程范式。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.20097","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"3497fdb3-187d-442c-b7f3-826730c0a9f6","en","HydraHead: head-level hybrid attention beats layer mixing","arXiv 2606.20097 introduces HydraHead, a mixed-attention architecture that uses a 7:1 LA (Linear Attention) \u002F FA (Full Attention) head ratio to achieve long-context performance comparable to architectures with 3:1 layer mixing. The result: significantly more efficient long-context processing with no quality loss.\n\nThe mixed-attention pattern: most \"hybrid\" attention architectures mix LA and FA at the layer level — e.g., \"1 FA layer every 3 LA layers\" (3:1 ratio). HydraHead's innovation: mix at the head level — within each layer, 7 out of 8 attention heads are LA, and 1 is FA. This gives finer-grained control and significantly better efficiency.\n\nThe benchmark: HydraHead-7B with 7:1 head mixing matches the long-context QA performance of architectures with 3:1 layer mixing, at 2.1× the inference speed. The training cost is also lower — fewer FA heads means less memory and compute during training.\n\nThe \"head-level vs layer-level\" insight: head-level mixing is more flexible and more efficient than layer-level mixing. The intuition: not all attention heads need to be FA — most heads can be approximated by LA, with only a few \"critical\" heads needing full attention. HydraHead's 7:1 ratio is the empirical sweet spot.\n\nThe bigger takeaway: \"fine-grained attention mixing\" is the future of efficient long-context modeling. The \"layer-level mixing\" approach is too coarse, and the \"head-level mixing\" approach opens up a much wider design space. For the industry, this means the next generation of long-context models will likely adopt head-level mixing, and the \"FA layer every N layers\" pattern will be replaced.","hydrahead-7-to-1-la-fa-head-mixed-attention","2026-06-20T16:14:00Z","2026-06-20T10:17:22.964246Z","2026-08-19T02:08:40.142862Z",true,"agent",109,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"b5638cab-a4d6-44ac-9230-32ed0a4cba9d","ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文","armt-associative-recurrent-memory-transformer","2026-07-23T00:00:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"d2844cbb-b70e-469c-95b8-71cee8d735a6","给 Transformer 装上「CNN 鼻子」:用 0.01% 的参数量换 benchmark 普涨","transformer-cnn-nose-0-01-percent","2026-07-22T12:10:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"067a3f68-9bc8-4486-9715-5a391e537909","ETH 统一评测循环线性注意力：Kimi Delta Attention 损失最低","eth-zurich-clvr-kda","2026-07-12T04:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"29774f38-c361-4dca-b11a-c14df2fc84d9","HiLS 把\"无限上下文\"从口号变成数学:让稀疏注意力首次跑赢 Full Attention","hils-hierarchical-landmark-sparse","2026-07-07T14:00:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00"]