[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-pytorch-sdpa-flash-attention-13":3,"news-related-4cbfe2a4-5b83-464d-bc5b-50acf224b1b0":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"4cbfe2a4-5b83-464d-bc5b-50acf224b1b0","torch.profiler 实测 SDPA：FlashAttention 13% 占用率真相","Hugging Face 工程团队 7 月 10 日发布「Profiling in PyTorch」系列第 3 篇,在 A100 上用 torch.profiler 把 PyTorch SDPA 的 math、efficient(xformers)、flash(FlashAttention-2)、cuDNN 四个后端逐一跑过,挖出几条「直觉与 trace 对不上」的发现。\n\n第一条:`masked_fill` 改成原地 `masked_fill_`,trace 里多出来的就是隐藏的 Memcpy。out-of-place 操作先复制再改写,原地版在 `no_grad` 下直接覆盖原 tensor 内存,对 logits 这种大张量既省 kernel 又省显存。改一行代码就砍掉整个 Memcpy,在多层 Transformer 里会叠加放大。\n\n第二条:`F.scaled_dot_product_attention` 一行顶五行,但 math 后端一次 forward 启 20 个 kernel,比手写还慢 3.7 倍。原因是 math 后端为数值稳定把输入升到 FP32、绕开 Tensor Core,而且每次调用都重建 causal mask。它的角色是参考实现,不是性能路径。\n\n第三条:四个 flash-style 后端都只跑一个 fused kernel,但 trace 上显示的「13% 占用率」很容易让人误判成性能差。flash 用 128 thread × 255 register\u002Fblock,每个 SM 只能装下两个 block,理论占用率约 13%——这正是 FlashAttention 把 `[seq, seq]` score 矩阵永远钉在片上、不写回 HBM 的代价,不是「它慢」。\n\ncuDNN 走另一条路:按输入 shape 现生成 kernel、用 `cuLaunchKernelEx` 驱动 API 直发、连 transpose 都省;代价是 plan 搜索全在 CPU,A100 这一组 shape 上 cuDNN(186.3µs)反而比 flash(146.8µs)慢。Profiler 报告「trace 变干净」并不等于「工作消失」,有时只是工作搬进了库函数看不到的地方。\n\n整个系列的方法论只有一句:**先猜,再打开 trace。** 所有值得分享的洞见——隐藏的 Memcpy、20 个 kernel 的 math backend、flash 错位的占用率、cuDNN 的 CPU 大柱子——都来自「猜测和 trace 对不上」的那一瞬间。","https:\u002F\u002Fhuggingface.co\u002Fblog\u002Ftorch-attention-profile","24d5c6c5-6573-4180-a1fd-f1459842d1af",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"7002537a-3eac-419b-b118-8ad37d31bcf3","en","SDPA under torch.profiler: FlashAttention's 13% occupancy","Hugging Face's engineering team released the 3rd installment of the \"Profiling in PyTorch\" series on July 10, using torch.profiler on A100 to walk through the four backends of PyTorch SDPA — math, efficient (xformers), flash (FlashAttention-2), and cuDNN — one by one, digging out several findings where \"intuition doesn't match the trace\". The first: changing `masked_fill` to in-place `masked_fill_`, and what shows up in the trace is a hidden Memcpy. Out-of-place operations copy first then modify, while the in-place version directly overwrites the original tensor's memory under `no_grad`, saving both kernel and memory for large tensors like logits. Changing one line of code cuts the entire Memcpy, and in multi-layer Transformers the effect compounds. The second: one line of `F.scaled_dot_product_attention` replaces five lines, but the math backend launches 20 kernels in one forward, 3.7× slower than handwritten. The reason is that the math backend promotes inputs to FP32 for numerical stability, bypassing Tensor Cores, and reconstructs the causal mask every call. Its role is the reference implementation, not the performance path. The third: all four flash-style backends only run one fused kernel, but the \"13% occupancy\" shown in the trace is easily misread as poor performance. flash uses 128 thread × 255 register\u002Fblock, each SM only fits two blocks, theoretical occupancy about 13% — this is exactly the cost of FlashAttention keeping the `[seq, seq]` score matrix pinned on-chip, never written back to HBM, not \"it's slow\". cuDNN takes a different path: it generates kernels on the fly based on input shape, uses the `cuLaunchKernelEx` driver API to directly launch, even saving transpose; the cost is that the plan search is all on CPU, and on A100 with this group of shapes cuDNN (186.3µs) is actually slower than flash (146.8µs). The Profiler's \"trace gets cleaner\" doesn't mean \"work disappears\" — sometimes the work just moves into a place in the library function you can't see. The entire series' methodology is one sentence: **guess first, then open the trace**. All the insights worth sharing — hidden Memcpy, 20-kernel math backend, flash's misaligned occupancy, cuDNN's CPU spike — come from that moment when \"guess and trace don't match\".","pytorch-sdpa-flash-attention-13","2026-07-11T08:01:00Z","2026-07-11T08:11:28.528111Z","2026-08-19T02:08:40.142862Z",true,"agent",81,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"070aef27-5fdb-4f0a-8b90-99afc1ea34fb","Jet-Long 用「动态双焦 RoPE」让 Qwen3 免训练扩到 128K,RULER 直接多涨 4.79 pp","jet-long-dynamic-dual-rope","2026-07-12T02:30:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"f0cab9bc-1b73-4362-80c7-f621be56ef5c","CARVE 把 GDN-2 的「记忆盲区」补上：用输出张量「白嫖」内容信号，1.3B 模型长上下文检索刷新 SOTA","carve-gdn2-content-aware-recurrent","2026-06-29T04:00:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"f63a58a9-85c9-406c-beef-0ba1cb0c6985","Taylor-Calibrate 把 Transformer 蒸馏成 GDN 的初始化做成系统级工程","taylor-calibrate-transformer-gdn-distill-88x","2026-06-21T14:30:00+00:00"]