[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-cohere-megakernel-north-mini-code-h100":3,"topics-all":38,"news-related-0fe9ceb8-6411-4924-8e02-8cee3665fc6f":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"0fe9ceb8-6411-4924-8e02-8cee3665fc6f","Cohere 开源 megakernel 推理引擎：单 CUDA 文件，H100 解码吃到 62% 带宽光速","Cohere 把 North Mini Code 的解码前向压进单个持久 CUDA kernel：单卡 H100 批 1 解码 292 tok\u002Fs、达带宽光速 62%，较 vLLM 快 1.58 倍；端到端快 1.25-1.41 倍，支持连续批处理与 OpenAI 端点，已开源。","自回归解码的每一毫秒里，H100 的大部分显存带宽其实都在空转。Cohere 刚开源的 megakernel serving 引擎给了一组扎眼的数字：vLLM 服务 North Mini Code（30B 参数、每 token 激活 3.3B）时，解码吞吐只有 185 tok\u002Fs，占 H100 内存带宽理论极限（Speed-of-Light，约 470 tok\u002Fs）的 39%；换成他们的 megakernel，同一张卡跑到 292 tok\u002Fs、62%——批大小 1 下解码比 vLLM 快 1.58 倍。\n\n## 解码是带宽游戏，不是算力游戏\n\nBF16 下，这个模型每个解码步要流过 6.6 GB 权重，8K 上下文再加约 0.5 GB KV cache，而 H100 的 HBM 带宽是 3.35 TB\u002Fs。传统 serving 栈把一次前向拆成上百个小 kernel 逐个发射：kernel 边界要等最慢的 SM 收齐、wave 量化浪费整波算力、算子间的伪依赖让权重流断断续续。Cohere 统计，这些停摆让典型推理引擎浪费了约 61% 的带宽。\n\n## 一个 persistent kernel 跑完整个解码步\n\nmegakernel 的做法是把整个前向压进单个持久 kernel：每个 SM 一个常驻线程块，从全局内存的任务列表领活，同步粒度从「整个 GPU」缩到「我真正依赖的生产者」。收益大头有三块：消 wave 量化（200 个 tile 摊到 132 个 SM，不再被迫凑整波）、拆伪依赖（O-proj 不必等全部 attention 组收齐）、权重预取（权重与激活无关，上一层 O-proj 收尾时就开始流 router\u002FQKV 的权重）。North Mini Code 的并行 transformer 层——attention 与 MoE 吃同一份归一化输入、出口处用融合 residual+RMSNorm 汇合——让空闲 SM 能被确定性回填。整套引擎是单个 CUDA 文件，16 个 opcode 覆盖完整解码图，没有编译器，没有新编程范式。\n\n## 从 demo 到完整 serving 系统\n\n以往 megakernel 工作多是 batch 1 的独立演示；Cohere 自称这是第一个围绕 decode megakernel 搭建的完整 serving 系统：连续批处理、paged attention、ragged 序列长度、带 tool calling 的 OpenAI 兼容端点，接上 OpenCode 就能写码。端到端比 vLLM v0.24 快 1.25–1.41 倍，加速保持到 256K 上下文，无可测精度损失。\n\n## 冷水与看点\n\n定位是 research release：单张 H100（sm_90a）、批大小上限 8、要求 CUDA 13+。思路源头是 Hazy Research 的 \"Look Ma, No Bubbles!\"（把 Llama-3.2-1B 前向融成单 kernel，吃到 78% 带宽）。真正值得记的有两条：其一，Cohere 称 megakernel「远比名声好写」，靠统一 ABI（3 个 warp group、32 个 int32 的任务描述符）把工程难度压了下来；其二，模型结构在给系统留口子——并行层设计正是激进回填的前提。当模型侧加速卷到边际收益递减，推理竞赛的下一程，可能落在 kernel 调度与模型-系统协同设计上。\n\n参考：https:\u002F\u002Fcohere.com\u002Fblog\u002Fmegakernels ；代码 https:\u002F\u002Fgithub.com\u002Fcohere-ai\u002Fcohere-megakernel","https:\u002F\u002Fcohere.com\u002Fblog\u002Fmegakernels","df9f8204-8e8d-4fce-8526-3c6fe8e6ae56",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"8e3ef7ba-e0a3-4ef9-a0ab-31b2761f7eba","en","Cohere megakernel engine: one CUDA file, 1.58x vLLM on H100","One persistent CUDA kernel runs North Mini Code's full decode: 292 tok\u002Fs at batch 1 on H100, 62% of speed-of-light, 1.58x vLLM.","Autoregressive decoding leaves most of an H100's memory bandwidth idle. Cohere's newly open-sourced megakernel serving engine posts stark numbers: vLLM serving North Mini Code (30B parameters, 3.3B active per token) decodes at 185 tok\u002Fs — 39% of the H100's Speed-of-Light (about 470 tok\u002Fs). Their megakernel reaches 292 tok\u002Fs, 62%, on the same card: 1.58x faster than vLLM at batch size 1.\n\n## Decode is a bandwidth game\n\nIn BF16, every decode step streams 6.6 GB of weights plus roughly 0.5 GB of KV cache at 8K context; the H100's HBM delivers 3.35 TB\u002Fs. A conventional serving stack launches a hundred small kernels per forward pass: kernel boundaries wait for the slowest SM, wave quantization wastes whole waves, and false dependencies keep the weight stream stuttering. Cohere counts those stalls as most of the 61% of bandwidth a typical inference engine leaves unused.\n\n## One persistent kernel for the whole decode step\n\nThe megakernel packs the entire forward pass into a single persistent kernel: one resident threadblock per SM pulls work from a task list in global memory, and the unit of synchronization shrinks from the whole GPU to the specific producers a task depends on. The big wins: wave quantization eliminated (200 tiles spread over 132 SMs no longer round up to whole waves), false dependencies dropped (O-proj starts once its KV group's attention lands), and weight prefetch (weights are activation-independent; router and QKV weights stream during the tail of the previous layer's O-proj). North Mini Code's parallel transformer layers — attention and MoE reading the same normalized input, rejoined by a fused residual add + RMSNorm — let idle SMs be deterministically backfilled. The whole engine is a single CUDA file, sixteen opcodes covering the full decode graph, no compiler, no new programming paradigm.\n\n## From demo to a full serving system\n\nEarlier megakernel work was mostly standalone demos at batch size 1; Cohere says this is the first fully fledged serving system built around a decode megakernel: continuous batching, paged attention, ragged sequence lengths, and an OpenAI-compatible endpoint with tool calling — point OpenCode at it and you can code. End-to-end it is 1.25-1.41x faster than vLLM v0.24, the margin holds out to 256K context, with no measurable accuracy loss.\n\n## Caveats and takeaways\n\nThis is a research release: single H100 (sm_90a), batch sizes up to 8, CUDA 13+. The lineage is Hazy Research's \"Look Ma, No Bubbles!\" (a Llama-3.2-1B forward pass fused into one kernel, hitting 78% of bandwidth). Two things worth remembering: Cohere claims megakernels are far easier to write than their reputation suggests, tamed by a uniform ABI (3 warp groups, 32-int32 task descriptors); and model structure is leaving room for the system — those parallel layers are precisely what enables aggressive backfill. As model-side acceleration hits diminishing returns, the next leg of the inference race may live in kernel scheduling and model-system co-design.\n\nReference: https:\u002F\u002Fcohere.com\u002Fblog\u002Fmegakernels ; code at https:\u002F\u002Fgithub.com\u002Fcohere-ai\u002Fcohere-megakernel","cohere-megakernel-north-mini-code-h100","2026-09-08T21:13:46Z","2026-09-08T21:13:51.651454Z","2026-09-08T21:13:51.651461Z",true,"agent",152,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"2638aeac-dc4d-4b73-b7fe-2b042015adee","OreoLook 开源:三层缓存把 AI 搜索搬进 8 核 CPU,重复问题 0.1 毫秒出答案","oreolook-three-layer-cpu-cache","2026-09-10T23:08:36+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"0c29e1ad-914a-4b79-a153-445c087acb03","被 LLM 抛弃的 dropout 翻身:Cerebras 称调好可省 25% 训练 FLOPs","dont-drop-dropout-layer-sparsity","2026-09-07T21:06:35+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"199cd4ef-f092-45a5-8635-91778dd2bce2","编译即训练：一句规约炼出 83.6% 准确率的本地神经函数，教师模型只用一次","compile-by-training-neural-functions","2026-09-04T23:08:03+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"7a6d28b6-65da-4a29-96b1-dedb9894de97","随机驱逐追平最强打分器:Salesforce 重写 KV Cache 压缩常识","random-attention-kv-cache-eviction","2026-09-04T19:08:26+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"44a035c8-b8a3-48e5-af4f-c76323dac7b5","RWKV7-G1j 13.3B 开源:不用注意力,每 token 推理成本是常数","rwkv7-g1j-13b-attention-free","2026-09-03T13:14:19+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"ff3b7c9f-e338-4af9-a562-4fda8b1bc929","vLLM 0.28 发布:584 项提交押注 Kimi K3,每卡省 17GB 显存","vllm-0-28-kimi-k3-optimization","2026-08-29T19:05:00+00:00"]