PTX (Parallel Thread Execution) is the lowest-level programmable interface CUDA developers can explicitly control. Every recent NVIDIA GPU generation adds new architecture-specific instructions at this layer — GMMA compute and UTMA payload movement on Hopper, the TCGEN05 tensor path on Blackwell. A portability-focused CUDA kernel can run perfectly correct on new hardware while leaving the defining capabilities of that hardware completely idle. High-performance kernels that actually harvest new silicon require PTX-level hand optimization, which is exactly the territory of scarce experts. PTXBench (arXiv:2608.17379), submitted August 18 by Kunle Olukotun group at Stanford together with CMU and RadixArk, puts the question squarely on the table: can LLMs do this work directly?

Three measurement layers: correct first, then instruction execution, then speed

Existing GPU kernel benchmarks (KernelBench and successors) mostly ask models to replace PyTorch operators with faster kernels and score end-to-end outcomes. That cannot isolate whether a model truly writes architecture-specific code — performance may come from generic CUDA, or simply from calling vendor libraries. PTXBench pushes the probe deeper with three layers:

  • Functional correctness: outputs aligned with reference implementations (torch.allclose, atol=rtol=1e-2);
  • Whether target instructions actually execute: static SASS inspection first, then Nsight Compute predicate-enabled thread counts for dynamic verification, ruling out false hits in dead code;
  • Performance: CUPTI timing, median over 50 iterations after 10 warmups, baselined against frontier libraries — cuBLAS v13.1.0 (GEMM), cuDNN v9.20.0 and FlashInfer v0.6.14 (attention).

Tests run on H100 and B200 across GEMM and MHA forward/backward plus causal variants. Models generate CUDA kernels with inline PTX from scratch inside the MiniPTXAgent multi-turn loop; including cuBLAS/cuDNN headers is marked wrong. Each task ships a 20k-30k token architecture knowledge pack — the ablation shows that without this context, models rarely use the requested PTX at all.

Results: forward is passable, backward collapses, Blackwell is harder

The core finding is blunt: architecture-specific PTX capability is deeply uneven, and no evaluated model consistently matches frontier libraries across the suite. Concrete numbers (H100, target-instruction metric, turn-0 correctness):

  • Claude Opus 4.8 hits 91.7% on GEMM at turn 0, reaching 94.8% within 8 turns — the most stable performer;
  • The same Claude scores just 8.3% on MHA backward at turn 0, needing all 8 turns to climb to 79.2%;
  • Gemini 3.1 Pro and GLM-5.2 both manage only 33.3% on GEMM at turn 0; the latter reaches just 5.2% on MHA backward within 8 turns;
  • On B200, everyone drops further: Gemini 3.1 Pro falls to 8.3% on GEMM turn 0, and attention forward reaches only 15.6% in 8 turns.

A subtler point: executing the target instruction does not mean competitive performance. Even with runtime-verified instruction execution, most implementations remain slower than frontier libraries. Using the new instructions and extracting their value are separated by an entire layer of scheduling engineering.

Fixit: turning a model own failures into training data

Beyond the benchmark, the team ran what they believe is the first controlled study of repair-conditioned SFT for architecture-specific CUDA/PTX generation, dubbed Fixit: start from failed kernels produced by the model being adapted (Qwen3.6-27B), have a repair teacher generate fixes that pass correctness checks, then have a reasoning teacher synthesize the chain from failure to repair.

The direction is clear: base Qwen3.6-27B produces zero correct GEMM kernels in any turn; after Fixit, turn 0 yields three correct ones, and six of seven later turns contain at least one. The SFT-vs-context comparison is telling: the base model cannot write correct MHA kernels even with expert guidance, while the Fixit version does so without guidance — SFT improves the underlying PTX capability itself, not just guidance comprehension. Meanwhile, retrieving repair notes alone produces zero correct kernels for the base model; only when the fixed kernel is also supplied does correctness jump — but that is essentially putting the answer in the prompt.

Limitations are stated plainly: adaptation used a single 27B model with modest LoRA datasets; workloads cover BF16 GEMM and attention on H100/B200 only. Cross-language transfer to Triton degrades Fixit correctness while improving peak speedup on causal variants (0.238x to 0.632x on MHA-Fwd-Causal).

So what

The value of PTXBench is not a leaderboard — it turns the question of whether LLMs can exploit rapidly evolving GPU architectures into an auditable testbed. For model vendors, it exposes the vacuum beneath coding benchmarks: writing a kernel that runs and writing one that saturates hardware are two different skills. For inference infrastructure teams, this architecture-specific capability directly determines migration cost as H100 gives way to B200 and beyond. Code is open-sourced at github.com/zhang677/PTXBench. The question worth watching: as repair-conditioned SFT scales from 27B to industrial post-training, will this fault line close — or does PTX remain the preserve of compilers and a handful of experts.

Source: https://arxiv.org/abs/2608.17379