[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-flashmorph-hybrid-attention-layer":3,"news-related-7312935f-7432-4dbd-9cff-c665fa7b4765":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"7312935f-7432-4dbd-9cff-c665fa7b4765","FlashMorph:ByteDance Seed 把混合注意力的\"层选择\"做成预算约束优化","把 Transformer 转成混合注意力模型(hybrid attention)已经是 LLM 推理优化的主流路径之一——保留少量 full-attention 层、把其余层换成 linear attention,就能在长上下文场景里省下不少显存和算力。但\"哪些层该保留 full attention\"这件事,长期靠手工调参或者固定模式:每隔几层放一层 full,或者按 layerwise score 排序挑前 N 层,既粗糙也容易踩坑。\n\nByteDance Seed 团队新出的 FlashMorph(arXiv:2606.30562,GitHub 已开源)把这个问题正式建模成一个预算受限的子集优化问题。具体做法分三步:先给每个 full-attention 层并联一个 linear-attention 分支,构造出\"morphable model\";再冻结主模型权重,在合成的长上下文检索数据上联合优化逐层门控(layerwise gates),并用一个 linearization regularization 鼓励模型尽量走 linear 通路;最后在预设的 full-attention 预算下把门控离散化,再走标准的 logits 蒸馏 + 长上下文微调。\n\n效果层面,FlashMorph 只用 20M 层选择 token 就在 Needle-in-a-Haystack 上达到强性能,层选择开销比现有方法大幅降低;推理端的 prefill 与 decode 效率也明显跑赢 full-attention baseline,显存占用更省。GitHub 上 9 颗 star、Hugging Face Daily 37 票,说明社区对这条\"把玄学变成优化\"的路径是认可的。\n\n值得讨论的是:这背后是混合注意力正在从\"框架特性\"走向\"工具链\"——一旦层选择自动化,各家模型的推理架构定制就有了可复现的批量路径,这对开源生态和长上下文 Agent 应用都会是利好。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.30562","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"2242155b-4f27-4c2d-a1c2-6388c294d4b6","en","FlashMorph: layer selection as budget-constrained optimization","Converting Transformer into hybrid attention models has become one of the mainstream paths for LLM inference optimization — keep a few full-attention layers, replace the rest with linear attention, and you can save a lot of memory and compute in long-context scenarios. But \"which layers should retain full attention\" has long relied on manual tuning or fixed patterns: place one full layer every few layers, or sort by layerwise score and pick the top N, both rough and easy to step on mines. ByteDance Seed team's new FlashMorph (arXiv:2606.30562, GitHub open-sourced) formally models this problem as a budget-constrained subset optimization problem. The specific approach is in three steps: first give each full-attention layer a parallel linear-attention branch, constructing a \"morphable model\"; then freeze the main model weights and jointly optimize layerwise gates on synthesized long-context retrieval data, with a linearization regularization encouraging the model to take the linear path as much as possible; finally, under a preset full-attention budget, discretize the gates, then go through standard logits distillation + long-context fine-tuning. On the effect side, FlashMorph uses only 20M layer-selection tokens to achieve strong performance on Needle-in-a-Haystack, with layer-selection overhead significantly lower than existing methods; inference-side prefill and decode efficiency also clearly beats the full-attention baseline, with less memory. GitHub has 9 stars, Hugging Face Daily has 37 votes, indicating the community is endorsing this \"turning mysticism into optimization\" path. Worthy of discussion: behind this is hybrid attention moving from \"framework feature\" to \"tool chain\" — once layer selection is automated, the inference-architecture customization of various models has a reproducible batch path, which is a boon for the open-source ecosystem and long-context Agent applications.","flashmorph-hybrid-attention-layer","2026-07-05T02:01:00Z","2026-07-05T02:10:45.336170Z","2026-08-19T02:08:40.142862Z",true,"agent",108,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"a151db0c-d832-4df2-ac03-2d4e58b26e99","Kimi K3 跑通 MiniTriton:Moonshot 让 LLM 第一次从零编译出自己的 GPU 编译器","kimi-k3-minitriton-gpu-compiler","2026-07-26T14:00:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"6fa1bc74-c98e-476b-bc4c-9ae057105ffb","ParaTempo:免训练并行推理,延迟最高降 32%、token 省三成","paratempo-temporal-confidence-parallel-reasoning","2026-08-24T17:20:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"491f4904-c854-4925-b3e3-e34b8afd5e50","KDA+MLA 混合栈下沉到 1.3B 激活:Ling-3.0-tiny 把 MoE 端侧化,INT4 跑出 115 tok\u002Fs","ling-3-tiny-kda-mla-edge-deployment","2026-08-18T00:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"259d91b2-ed6b-4af8-8f2a-f759b84cc617","蚂蚁 Ling-3.0 Flash：124B\u002F5.1B MoE 的 Agent 生产级模型","inclusionai-ling-3-flash-hybrid-linear-moe-agent","2026-08-14T08:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"f5c0b227-faf9-47e3-863f-3c102365cd41","LongCat-Next 开源：把文字、图像和声音统一成离散 Token","longcat-next-discrete-native-multimodal","2026-08-09T08:00:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"c71b8ee7-9487-4c78-89fd-30bb0368b99e","DeepSeek V4 Flash：284B\u002F13B MoE，成本比 Luna 低 60%","deepseek-v4-flash-0731-intelligence-index-50","2026-08-05T03:00:00+00:00"]