Hugging Face's engineering team released the 3rd installment of the "Profiling in PyTorch" series on July 10, using torch.profiler on A100 to walk through the four backends of PyTorch SDPA — math, efficient (xformers), flash (FlashAttention-2), and cuDNN — one by one, digging out several findings where "intuition doesn't match the trace". The first: changing masked_fill to in-place masked_fill_, and what shows up in the trace is a hidden Memcpy. Out-of-place operations copy first then modify, while the in-place version directly overwrites the original tensor's memory under no_grad, saving both kernel and memory for large tensors like logits. Changing one line of code cuts the entire Memcpy, and in multi-layer Transformers the effect compounds. The second: one line of F.scaled_dot_product_attention replaces five lines, but the math backend launches 20 kernels in one forward, 3.7× slower than handwritten. The reason is that the math backend promotes inputs to FP32 for numerical stability, bypassing Tensor Cores, and reconstructs the causal mask every call. Its role is the reference implementation, not the performance path. The third: all four flash-style backends only run one fused kernel, but the "13% occupancy" shown in the trace is easily misread as poor performance. flash uses 128 thread × 255 register/block, each SM only fits two blocks, theoretical occupancy about 13% — this is exactly the cost of FlashAttention keeping the [seq, seq] score matrix pinned on-chip, never written back to HBM, not "it's slow". cuDNN takes a different path: it generates kernels on the fly based on input shape, uses the cuLaunchKernelEx driver API to directly launch, even saving transpose; the cost is that the plan search is all on CPU, and on A100 with this group of shapes cuDNN (186.3µs) is actually slower than flash (146.8µs). The Profiler's "trace gets cleaner" doesn't mean "work disappears" — sometimes the work just moves into a place in the library function you can't see. The entire series' methodology is one sentence: guess first, then open the trace. All the insights worth sharing — hidden Memcpy, 20-kernel math backend, flash's misaligned occupancy, cuDNN's CPU spike — come from that moment when "guess and trace don't match".