[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sglang-v0-5-18-cold-start-2-38x":3,"news-related-c94bdf86-5de9-49fe-8c98-0f5c47611bfe":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","开源推理框架 SGLang 发布 v0.5.18:合并 710 个 PR,权重加载与 CUDA 图捕获重叠执行,让 Qwen3-32B 冷启动从 84.8 秒降到 35.6 秒,还带来 TP LMHead all-to-all 通信优化、AMD MI355X 内核大幅提速与 7 个新模型支持。","如果你管过线上大模型服务,大概率遇到过这种尴尬:流量高峰要紧急扩容,新拉起的推理实例却卡在加载权重和捕获 CUDA 图上,一分多钟才 ready。SGLang 8 月 22 日发布的 v0.5.18 把这个问题当成了头号目标——合并 212 位贡献者的 710 个 PR,把冷启动时间砍掉近三分之二。\n\n## 冷启动:84.8 秒压到 35.6 秒\n\n核心改动叫 overlapped checkpoint staging:权重分页从存储加载,不再和 CUDA graph capture 串行等待,而是重叠执行。Qwen3-32B 在 H100 上,启动比串行加预取快 8.6-11.7%,对比默认路径快 2.38 倍——84.8 秒压到 35.6 秒,开启只需一个参数:--startup-weight-load-mode overlap。对弹性扩缩容和 Spot 实例场景,这是实打实的成本项。\n\n## 解码路径上的微雕\n\nTP LMHead 的 allgather 加 scatter 两步,在纯 DP 注意力下合并成一次 all-to-all:DeepSeek-V4-Pro 在 B200 上解码,LMHead 耗时从 320 微秒降到 169 微秒,TPOT 从 36.97 毫秒改善到 35.67 毫秒。单步看只有一毫秒出头,乘上每秒上万次解码步,累积收益可观。另外非融合 allreduce 现在复用 FlashInfer MNNVL workspace,DeepSeek-V4-Flash TP4 在 Blackwell 小 batch 场景最高再涨 6.9%。\n\n## AMD 阵营的意外惊喜\n\nMI355X 用户这轮礼物很厚:Kimi K3 换上分组头 MLA verify 内核,吞吐 1.37-1.77 倍;GLM-5.2 在 gfx950 上吞吐 +14%、TPOT 降 12.5%,短上下文走稠密回退让 TTFT 再降 22%-43%。更有意思的是 NVFP4 转 MXFP4 在线重量化:加载时直接把英伟达格式权重转成 AMD 能跑的 MXFP4,不持有全精度副本,在 GLM-5.1、Kimi-K2.6、Qwen3.5-397B 等五个模型上精度保留 97.5%-100.2%。英伟达 checkpoint 开箱跑 AMD,以前要自己折腾半天。\n\n## 扩散模型与升级注意\n\nSGLang-Diffusion 把 breakable CUDA graph 铺到更多模型:LTX-2 在 H200 端到端从 10.75 秒降到 6.90 秒,SANA 1024px 降 26%。新支持 Muse Glimmer、SANA-Video、LTX-2.5 等 7 个模型。升级有两个坑:CUDA 栈升到 torch 2.13.0;编译内核缓存统一收进 SGLANG_CACHE_DIR,升级后首启会重编译一次,预热镜像的团队记得提前迁移目录。\n\n## 所以呢\n\n模型层每周都有新旗舰,但推理框架决定这些模型落地时的真实成本。v0.5.18 的信号很清晰:冷启动、通信原语、跨硬件移植,正在成为 vLLM 与 SGLang 竞争的主战场。下次服务扩容慢了半拍,不妨看看是不是框架该升了。\n\n参考:SGLang v0.5.18 Release Notes(https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\u002Freleases\u002Ftag\u002Fv0.5.18)","https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\u002Freleases\u002Ftag\u002Fv0.5.18","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"0b33242d-4eac-4037-b809-83cf7fbbcdee","en","SGLang v0.5.18 Ships: LLM Cold Start 2.38x Faster Across 710 PRs","Open-source inference framework SGLang releases v0.5.18: 710 PRs merged, overlapping weight loading with CUDA graph capture cuts Qwen3-32B cold start from 84.8s to 35.6s, plus TP LMHead all-to-all communication optimization, major AMD MI355X kernel gains, and 7 new models.","If you have ever run production LLM serving, you know the pain: traffic spikes demand urgent scaling, but freshly launched inference instances sit there loading weights and capturing CUDA graphs for over a minute before they are ready. SGLang's v0.5.18, released on August 22, made this problem target number one — merging 710 PRs from 212 contributors and cutting cold start time by nearly two-thirds.\n\n## Cold Start: 84.8 Seconds Down to 35.6\n\nThe core change is called overlapped checkpoint staging: checkpoint pages now stream from storage while CUDA graph capture runs, instead of waiting in series. On the official benchmark, Qwen3-32B on H100 starts 8.6-11.7% faster than serial-with-prefetch, and 2.38x faster than the plain default — 84.8 seconds compressed to 35.6. It is opt-in with a single flag: --startup-weight-load-mode overlap. For elastic autoscaling and spot-instance workloads, this is a direct cost line item.\n\n## Micro-Carving the Decode Path\n\nThe TP LMHead's allgather-plus-scatter pair collapses into a single all-to-all under pure-DP attention: on DeepSeek-V4-Pro B200 decode, LMHead time drops from 320 microseconds to 169, and TPOT improves from 36.97ms to 35.67ms. A millisecond per step sounds small, but multiplied by tens of thousands of decode steps per second, the accumulated gain is real. Non-fused allreduce sites now also reuse the FlashInfer MNNVL workspace instead of falling back to NCCL, adding up to +6.9% for DeepSeek-V4-Flash TP4 at small batches on Blackwell.\n\n## An Unexpected Gift for AMD\n\nMI355X users get a thick package this cycle: Kimi K3 gains a grouped-head MLA verify kernel with 1.37-1.77x throughput; GLM-5.2 on gfx950 gets fp8 MLA absorbed bmm at +14% throughput and -12.5% TPOT, with a dense short-context fallback cutting TTFT another 22-43%. The most interesting piece is online NVFP4-to-MXFP4 requantization: at load time, NVIDIA-format weights are converted directly to AMD-runnable MXFP4 without ever holding a full-precision copy, retaining 97.5%-100.2% accuracy across GLM-5.1, Kimi-K2.6, Qwen3.5-397B and three others. Running NVIDIA checkpoints on AMD out of the box used to be an afternoon of pain.\n\n## Diffusion Models and Upgrade Notes\n\nSGLang-Diffusion extends breakable CUDA graphs to more models: LTX-2 on H200 drops end-to-end from 10.75s to 6.90s, SANA at 1024px gets -26%. Seven new models are supported, including Muse Glimmer, SANA-Video, and LTX-2.5. Two upgrade gotchas: the CUDA stack moves to torch 2.13.0, and every compiled-kernel cache consolidates under SGLANG_CACHE_DIR — the first launch after upgrading recompiles once, so teams with pre-warmed images should migrate directories ahead of time.\n\n## So What\n\nNew frontier models ship every week, but inference frameworks decide their real cost at deployment. The signal from v0.5.18 is clear: cold start, communication primitives, and cross-hardware portability are now the main battleground between vLLM and SGLang. Next time your service scales out too slowly, check whether the framework needs an upgrade.\n\nReference: SGLang v0.5.18 Release Notes (https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\u002Freleases\u002Ftag\u002Fv0.5.18)","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00Z","2026-08-24T23:10:35.484732Z","2026-08-24T23:10:35.484740Z",true,"agent",81,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"92433e6b-113a-4ada-af77-fbb8995a9850","LFM2.5-DSpark 开源:300M 草稿模型让端侧推理快 2.87 倍,输出零损耗","lfm2-5-dspark-draft-models","2026-08-21T21:10:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"894787b7-8799-4c22-8e20-5e04db8f7181","DeepSeek V4 GA 把分时电价塞进大模型 API:CSA + HCA 撑起 1M 上下文,峰值翻倍只是商业答案","deepseek-v4-ga-time-of-use-pricing","2026-07-24T02:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"518d0be2-1220-4920-98fd-dee9df27a43d","VIDRAFT VKUE 把同一份 34.7B 稀疏 MoE 权重从 B200 跑到裸 CPU","vidraft-vkue-cpu-moe","2026-07-12T10:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":64},"f8a33525-3846-46e2-9f88-05be407dbe2f","Liquid AI 用 Antidoom 把推理模型的 doom loop 压到 1%：一次训练只改 1 个 token","liquid-ai-antidoom"]