[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-ptxbench-llm-ptx-gpu-kernel-benchmark":3,"news-related-4c54e7dc-46f6-4d0e-86c9-278995cf0da8":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4c54e7dc-46f6-4d0e-86c9-278995cf0da8","Stanford PTXBench:让 LLM 裸写 H100\u002FB200 PTX kernel,没有一个模型全过关","Stanford 发布 PTXBench:专测 LLM 能否用 Hopper\u002FBlackwell 专属 PTX 从零写出正确 GPU kernel。结果极不均衡:GEMM 尚可,attention backward 集体跳水,B200 更差,没有模型能稳定追平 cuBLAS 等前沿库;Fixit SFT 让 Qwen3.6-27B 从零正确到能写对。","PTX(Parallel Thread Execution)是 CUDA 开发者能显式控制的最低级可编程接口。最近几代 NVIDIA GPU 每次迭代都在这层加入新的架构专属指令——Hopper 的 GMMA 计算与 UTMA 数据搬运、Blackwell 的 TCGEN05 tensor 路径。一个追求可移植的 CUDA kernel 可以在新硬件上跑得完全正确,却把新一代硬件的定义性能力全部闲置。高性能 kernel 要吃到新卡的算力,绕不开 PTX 级的手工优化,而这恰恰是稀缺专家的领地。Stanford 的 Kunle Olukotun 组联合 CMU、RadixArk 在 8 月 18 日提交的 PTXBench(arXiv:2608.17379)就把问题摆上了台面:LLM 能不能直接干这活?\n\n## 三层指标:先问对不对,再问指令跑没跑,最后问快不快\n\n现有 GPU kernel 基准(如 KernelBench)大多让模型把 PyTorch 算子替换成更快的 kernel,只看端到端结果。但这测不出模型是否真的会写架构专属代码——性能可能来自通用 CUDA,或者干脆是调了厂商库。PTXBench 把探针扎得更深,拆成三层:\n\n- **功能正确性**:输出与参考实现对齐(torch.allclose,atol=rtol=1e-2);\n- **目标指令是否真的执行**:先静态查 SASS 汇编里有没有目标指令族,再用 Nsight Compute 取 predicate-enabled 线程数做动态验证,排除死代码里的假命中;\n- **性能**:CUPTI 计时,50 次迭代取中位数(10 次 warmup),基线是 cuBLAS v13.1.0(GEMM)、cuDNN v9.20.0 与 FlashInfer v0.6.14(attention)这些前沿库。\n\n测试在 H100 和 B200 上进行,覆盖 GEMM 与 MHA 前向\u002F反向及 causal 变体。模型在 MiniPTXAgent 多轮循环里从零生成带内联 PTX 的 kernel,禁止 include cuBLAS\u002FcuDNN 头文件,靠结构化执行反馈迭代修改。每个任务附 20k–30k token 的架构知识包;消融实验显示,没有这份上下文,模型基本不会去用被要求的 PTX。\n\n## 结果:forward 尚可,backward 跳水,Blackwell 更难\n\n论文的核心结论很直白:架构专属 PTX 能力严重不均衡,且没有任何一个被测模型能在全套件上稳定追平前沿库。几个具体数字(H100,目标指令口径,首轮 turn 正确率):\n\n- **Claude Opus 4.8** 在 GEMM 上首轮 91.7%,8 轮内到 94.8%,全场最稳;\n- 同一个 Claude,在 MHA backward 首轮只有 8.3%,要 8 轮才爬到 79.2%;\n- **Gemini 3.1 Pro** 与 **GLM-5.2** 的 GEMM 首轮都只有 33.3%,后者的 MHA backward 8 轮内仅 5.2%;\n- 换到 B200,全员进一步下滑:Gemini 3.1 Pro 的 GEMM 首轮掉到 8.3%,attention 前向 8 轮也只有 15.6%。\n\n更微妙的一点:目标指令执行了,不代表性能有竞争力。即便验证了 kernel 在运行时真的跑到了指定指令,大部分实现仍然慢于前沿库。会用新指令,和用出新指令的价值,中间隔着一整层调度工程。\n\n## Fixit:让模型把自己踩过的坑变成训练数据\n\nBenchmark 之外,团队还做了(据其所知)首个针对特定架构 CUDA\u002FPTX 生成的修复条件 SFT 受控研究,起名 Fixit:从被适配模型(Qwen3.6-27B)自己产生的失败 kernel 出发,repair teacher 生成通过正确性检查的修复版本,reasoning teacher 再合成从失败到修复的推理链,拼成监督数据。\n\n效果方向明确:基础版 Qwen3.6-27B 在 GEMM 上任何一轮都没有正确 kernel,Fixit 训练后第 0 轮就有 3 个正确,后续 7 轮里 6 轮至少 1 个正确。更有意思的是 SFT 与上下文监督的对比:基础模型即便配上专家指导也写不对 MHA kernel,而 Fixit 版不带指导也能写对——说明 SFT 提升的是底层 PTX 能力本身,不只是读懂指导的能力。反过来看,给基础模型检索修复笔记毫无用处,只有连修复后的 kernel 一起给才大幅起效——但那基本等于把答案放进 prompt。\n\n局限作者也摆明:适配实验只用了一个 27B 模型加适度的 LoRA 数据集;工作负载限于 H100\u002FB200 上的 BF16 GEMM 与 attention。跨语言迁移到 Triton 时,Fixit 版正确率反而下降,只是峰值加速比明显改善(causal 前向从 0.238× 提到 0.632×)。\n\n## 所以呢\n\nPTXBench 的价值不在排行榜,在于把 LLM 能否利用快速演进的 GPU 架构变成一个可审计的测试床。对模型厂商,它指出了 coding benchmark 之下的真空地带:会写能跑的 kernel 和会写吃满硬件的 kernel 是两种能力。对推理基础设施团队,当 H100 换 B200、B200 再换下一代,这种架构专属能力直接决定迁移成本。代码在 GitHub 开源(github.com\u002Fzhang677\u002FPTXBench)。下一个值得盯的问题:当修复条件 SFT 从 27B 规模走向工业级后训练,这条断层会不会被填平——还是说,PTX 终究是编译器和少数专家的自留地。\n\n原文: https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.17379","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.17379","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":19,"name":20,"slug":20,"description":14,"color":14},"e0d31e94-ce47-4c8f-831c-d3d2926d42f3","hardware",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3f3351c5-c197-4f67-a068-a4894d0270e9","en","Stanford PTXBench: LLMs Writing Raw H100\u002FB200 PTX Kernels — No Model Passes Everything","Stanford releases PTXBench, a benchmark testing whether LLMs can write correct GPU kernels from scratch using Hopper\u002FBlackwell-specific PTX. Results are sharply uneven: GEMM is passable, attention backward collapses, B200 is worse across the board, and no model consistently matches frontier libraries like cuBLAS. Fixit SFT lifts Qwen3.6-27B from zero correct kernels to working ones.","PTX (Parallel Thread Execution) is the lowest-level programmable interface CUDA developers can explicitly control. Every recent NVIDIA GPU generation adds new architecture-specific instructions at this layer — GMMA compute and UTMA payload movement on Hopper, the TCGEN05 tensor path on Blackwell. A portability-focused CUDA kernel can run perfectly correct on new hardware while leaving the defining capabilities of that hardware completely idle. High-performance kernels that actually harvest new silicon require PTX-level hand optimization, which is exactly the territory of scarce experts. PTXBench (arXiv:2608.17379), submitted August 18 by Kunle Olukotun group at Stanford together with CMU and RadixArk, puts the question squarely on the table: can LLMs do this work directly?\n\n## Three measurement layers: correct first, then instruction execution, then speed\n\nExisting GPU kernel benchmarks (KernelBench and successors) mostly ask models to replace PyTorch operators with faster kernels and score end-to-end outcomes. That cannot isolate whether a model truly writes architecture-specific code — performance may come from generic CUDA, or simply from calling vendor libraries. PTXBench pushes the probe deeper with three layers:\n\n- **Functional correctness**: outputs aligned with reference implementations (torch.allclose, atol=rtol=1e-2);\n- **Whether target instructions actually execute**: static SASS inspection first, then Nsight Compute predicate-enabled thread counts for dynamic verification, ruling out false hits in dead code;\n- **Performance**: CUPTI timing, median over 50 iterations after 10 warmups, baselined against frontier libraries — cuBLAS v13.1.0 (GEMM), cuDNN v9.20.0 and FlashInfer v0.6.14 (attention).\n\nTests run on H100 and B200 across GEMM and MHA forward\u002Fbackward plus causal variants. Models generate CUDA kernels with inline PTX from scratch inside the MiniPTXAgent multi-turn loop; including cuBLAS\u002FcuDNN headers is marked wrong. Each task ships a 20k-30k token architecture knowledge pack — the ablation shows that without this context, models rarely use the requested PTX at all.\n\n## Results: forward is passable, backward collapses, Blackwell is harder\n\nThe core finding is blunt: architecture-specific PTX capability is deeply uneven, and no evaluated model consistently matches frontier libraries across the suite. Concrete numbers (H100, target-instruction metric, turn-0 correctness):\n\n- **Claude Opus 4.8** hits 91.7% on GEMM at turn 0, reaching 94.8% within 8 turns — the most stable performer;\n- The same Claude scores just 8.3% on MHA backward at turn 0, needing all 8 turns to climb to 79.2%;\n- **Gemini 3.1 Pro** and **GLM-5.2** both manage only 33.3% on GEMM at turn 0; the latter reaches just 5.2% on MHA backward within 8 turns;\n- On B200, everyone drops further: Gemini 3.1 Pro falls to 8.3% on GEMM turn 0, and attention forward reaches only 15.6% in 8 turns.\n\nA subtler point: executing the target instruction does not mean competitive performance. Even with runtime-verified instruction execution, most implementations remain slower than frontier libraries. Using the new instructions and extracting their value are separated by an entire layer of scheduling engineering.\n\n## Fixit: turning a model own failures into training data\n\nBeyond the benchmark, the team ran what they believe is the first controlled study of repair-conditioned SFT for architecture-specific CUDA\u002FPTX generation, dubbed Fixit: start from failed kernels produced by the model being adapted (Qwen3.6-27B), have a repair teacher generate fixes that pass correctness checks, then have a reasoning teacher synthesize the chain from failure to repair.\n\nThe direction is clear: base Qwen3.6-27B produces zero correct GEMM kernels in any turn; after Fixit, turn 0 yields three correct ones, and six of seven later turns contain at least one. The SFT-vs-context comparison is telling: the base model cannot write correct MHA kernels even with expert guidance, while the Fixit version does so without guidance — SFT improves the underlying PTX capability itself, not just guidance comprehension. Meanwhile, retrieving repair notes alone produces zero correct kernels for the base model; only when the fixed kernel is also supplied does correctness jump — but that is essentially putting the answer in the prompt.\n\nLimitations are stated plainly: adaptation used a single 27B model with modest LoRA datasets; workloads cover BF16 GEMM and attention on H100\u002FB200 only. Cross-language transfer to Triton degrades Fixit correctness while improving peak speedup on causal variants (0.238x to 0.632x on MHA-Fwd-Causal).\n\n## So what\n\nThe value of PTXBench is not a leaderboard — it turns the question of whether LLMs can exploit rapidly evolving GPU architectures into an auditable testbed. For model vendors, it exposes the vacuum beneath coding benchmarks: writing a kernel that runs and writing one that saturates hardware are two different skills. For inference infrastructure teams, this architecture-specific capability directly determines migration cost as H100 gives way to B200 and beyond. Code is open-sourced at github.com\u002Fzhang677\u002FPTXBench. The question worth watching: as repair-conditioned SFT scales from 27B to industrial post-training, will this fault line close — or does PTX remain the preserve of compilers and a handful of experts.\n\nSource: https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.17379","ptxbench-llm-ptx-gpu-kernel-benchmark","2026-08-19T14:00:00Z","2026-08-19T13:12:35.399031Z","2026-08-19T13:12:35.399040Z",true,"agent",107,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"7ba15299-8bee-4039-8bc7-dbb58754b562","SWE-bench Science:最强 Claude Code 修科学代码也不及格,四类失败模式被拆解","swe-bench-science-benchmark","2026-08-21T13:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"5b76d928-8896-4bf8-8edd-e195ecf0094a","Ornith-1.5自报跑分赢了Claude,独立复测翻车了","ornith-1-5-benchmark-reality-check","2026-08-20T17:10:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"e846f1c6-4644-4f84-a664-83ec82734210","Meta Muse Spark 1.2 与 Muse Code 把「1.2 → 编程」的推理效率推回前沿","meta-muse-spark-12-coding-agent-54-index","2026-08-05T08:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"9382e481-e16b-4925-a0d1-3b24cd8ba22a","OpenAI 复审自家推荐的 SWE-Bench Pro:731 道题里约三成是「坏题」,榜单狂欢该降温","openai-swe-bench-pro-audit","2026-07-13T00:11:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"abccd085-d910-4846-aae1-5f2136238f3b","别怪大模型:Coding Agent 真正的瓶颈在「脚手架」——Queen's 35 版本控制变量实证","coding-agent-scaffold-bottleneck","2026-07-04T03:55:25+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"c4acfb32-44d4-45a9-ac27-ec9fff4e1eca","Mistral 开源 Leanstral 1.5:6B 激活参数刷新形式化推理 SOTA","mistral-leanstral-1-5","2026-07-04T00:30:00+00:00"]