When the attention mechanism becomes the LLM inference bottleneck, hardware architecture evolution is providing a new answer. NVIDIA Blackwell brings larger Tensor Cores, but the SFU responsible for exponential operations hasn't been upgraded in sync — softmax's exp() has for the first time become as expensive as matrix multiplication. FlashAttention-4 solves this with Tile-level "double buffering": alternating matrix multiplication with exponential operations, maximizing GPU utilization.
A deeper change lies in memory access. Blackwell introduces TMEM (Tensor Memory), fully asynchronizing data movement with matrix operations. The kernel must be deeply pipelined — some warps handle synchronous softmax, others schedule asynchronous loading. The PyTorch team partnered with Tri Dao to refactor FA4 as the FlexAttention backend, allowing custom attention variants to approach hardware limits.
Notably CuTeDSL — NVIDIA used Python DSL to rewrite CUTLASS core, bringing JIT-style Attention pipelines to production-grade implementation for the first time. PyTorch automatically generates CuTeDSL code through Inductor, letting users call FA4-level optimization without writing CUDA.
This means: the engineering path to longer context and lower-cost inference is now clear.