Autoregressive decoding leaves most of an H100's memory bandwidth idle. Cohere's newly open-sourced megakernel serving engine posts stark numbers: vLLM serving North Mini Code (30B parameters, 3.3B active per token) decodes at 185 tok/s — 39% of the H100's Speed-of-Light (about 470 tok/s). Their megakernel reaches 292 tok/s, 62%, on the same card: 1.58x faster than vLLM at batch size 1.
Decode is a bandwidth game
In BF16, every decode step streams 6.6 GB of weights plus roughly 0.5 GB of KV cache at 8K context; the H100's HBM delivers 3.35 TB/s. A conventional serving stack launches a hundred small kernels per forward pass: kernel boundaries wait for the slowest SM, wave quantization wastes whole waves, and false dependencies keep the weight stream stuttering. Cohere counts those stalls as most of the 61% of bandwidth a typical inference engine leaves unused.
One persistent kernel for the whole decode step
The megakernel packs the entire forward pass into a single persistent kernel: one resident threadblock per SM pulls work from a task list in global memory, and the unit of synchronization shrinks from the whole GPU to the specific producers a task depends on. The big wins: wave quantization eliminated (200 tiles spread over 132 SMs no longer round up to whole waves), false dependencies dropped (O-proj starts once its KV group's attention lands), and weight prefetch (weights are activation-independent; router and QKV weights stream during the tail of the previous layer's O-proj). North Mini Code's parallel transformer layers — attention and MoE reading the same normalized input, rejoined by a fused residual add + RMSNorm — let idle SMs be deterministically backfilled. The whole engine is a single CUDA file, sixteen opcodes covering the full decode graph, no compiler, no new programming paradigm.
From demo to a full serving system
Earlier megakernel work was mostly standalone demos at batch size 1; Cohere says this is the first fully fledged serving system built around a decode megakernel: continuous batching, paged attention, ragged sequence lengths, and an OpenAI-compatible endpoint with tool calling — point OpenCode at it and you can code. End-to-end it is 1.25-1.41x faster than vLLM v0.24, the margin holds out to 256K context, with no measurable accuracy loss.
Caveats and takeaways
This is a research release: single H100 (sm_90a), batch sizes up to 8, CUDA 13+. The lineage is Hazy Research's "Look Ma, No Bubbles!" (a Llama-3.2-1B forward pass fused into one kernel, hitting 78% of bandwidth). Two things worth remembering: Cohere claims megakernels are far easier to write than their reputation suggests, tamed by a uniform ABI (3 warp groups, 32-int32 task descriptors); and model structure is leaving room for the system — those parallel layers are precisely what enables aggressive backfill. As model-side acceleration hits diminishing returns, the next leg of the inference race may live in kernel scheduling and model-system co-design.
Reference: https://cohere.com/blog/megakernels ; code at https://github.com/cohere-ai/cohere-megakernel