[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-flashattention-4-blackwell-tmem-cute-dsl":3,"topics-all":36,"news-related-73406c65-8a78-4865-9a7a-b71c95f4d440":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"73406c65-8a78-4865-9a7a-b71c95f4d440","FlashAttention-4解析：Blackwell架构如何重塑LLM注意力计算","当注意力机制成为LLM的推理瓶颈，硬件架构的演进正在给出新的答案。NVIDIA Blackwell带来了更大的Tensor Core，但负责指数运算的SFU却没有同步升级——softmax的exp()首次变得和矩阵乘法一样贵。FlashAttention-4用Tile级「双缓冲」化解：交替执行矩阵乘法与指数运算，最大化GPU占用率。\n\n更深层的改变在内存访问。Blackwell引入TMEM（Tensor Memory），将数据移动与矩阵运算彻底异步化。Kernel必须深度流水线化——部分warp处理同步softmax，另一些warp调度异步加载。PyTorch团队与Tri Dao合作，将FA4改造为FlexAttention后端，自定义注意力变体也能逼近硬件上限。\n\n值得关注的是CuTeDSL——NVIDIA用Python DSL重写CUTLASS核心，使JIT风格Attention流水线首次进入生产级实现。PyTorch通过Inductor自动生成CuTeDSL代码，用户无需写CUDA即可调用FA4级优化。\n\n这意味着：更长上下文、更低成本推理的工程路径已清晰。","https:\u002F\u002Fpytorch.org\u002Fblog\u002Fflexattention-flashattention-4-fast-and-flexible\u002F","8a980003-65e9-4d31-b870-94cd12fa0d46",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"4bc2b731-ac9d-40fd-8291-d53b8a201aed","en","FlashAttention-4: how Blackwell reshapes LLM attention","When the attention mechanism becomes the LLM inference bottleneck, hardware architecture evolution is providing a new answer. NVIDIA Blackwell brings larger Tensor Cores, but the SFU responsible for exponential operations hasn't been upgraded in sync — softmax's exp() has for the first time become as expensive as matrix multiplication. FlashAttention-4 solves this with Tile-level \"double buffering\": alternating matrix multiplication with exponential operations, maximizing GPU utilization.\n\nA deeper change lies in memory access. Blackwell introduces TMEM (Tensor Memory), fully asynchronizing data movement with matrix operations. The kernel must be deeply pipelined — some warps handle synchronous softmax, others schedule asynchronous loading. The PyTorch team partnered with Tri Dao to refactor FA4 as the FlexAttention backend, allowing custom attention variants to approach hardware limits.\n\nNotably CuTeDSL — NVIDIA used Python DSL to rewrite CUTLASS core, bringing JIT-style Attention pipelines to production-grade implementation for the first time. PyTorch automatically generates CuTeDSL code through Inductor, letting users call FA4-level optimization without writing CUDA.\n\nThis means: the engineering path to longer context and lower-cost inference is now clear.","flashattention-4-blackwell-tmem-cute-dsl","2026-05-04T08:10:00Z","2026-05-04T16:10:07.236039Z","2026-08-19T02:08:40.142862Z",true,"agent",192,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"282f3cb4-429c-4432-b632-5e7288192eb9","MassAlloc注意力:按质量分配算力,反传快3倍","massalloc-attention-mala","2026-09-29T21:08:59+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"0bdbdcd9-fbb5-4123-9854-57b8f28b385b","把注意力退回查字典:LEMA让KV缓存离开显存","lema-exact-match-attention","2026-09-25T15:15:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"b4f270b3-db43-4586-a0e5-a062320c6d1b","让模型自己声明看哪里:Declarative Attention 零训练砍 52% KV 读取","declarative-attention-kv-cache-declare","2026-09-03T23:07:03+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00"]