If you have ever run production LLM serving, you know the pain: traffic spikes demand urgent scaling, but freshly launched inference instances sit there loading weights and capturing CUDA graphs for over a minute before they are ready. SGLang's v0.5.18, released on August 22, made this problem target number one — merging 710 PRs from 212 contributors and cutting cold start time by nearly two-thirds.

Cold Start: 84.8 Seconds Down to 35.6

The core change is called overlapped checkpoint staging: checkpoint pages now stream from storage while CUDA graph capture runs, instead of waiting in series. On the official benchmark, Qwen3-32B on H100 starts 8.6-11.7% faster than serial-with-prefetch, and 2.38x faster than the plain default — 84.8 seconds compressed to 35.6. It is opt-in with a single flag: --startup-weight-load-mode overlap. For elastic autoscaling and spot-instance workloads, this is a direct cost line item.

Micro-Carving the Decode Path

The TP LMHead's allgather-plus-scatter pair collapses into a single all-to-all under pure-DP attention: on DeepSeek-V4-Pro B200 decode, LMHead time drops from 320 microseconds to 169, and TPOT improves from 36.97ms to 35.67ms. A millisecond per step sounds small, but multiplied by tens of thousands of decode steps per second, the accumulated gain is real. Non-fused allreduce sites now also reuse the FlashInfer MNNVL workspace instead of falling back to NCCL, adding up to +6.9% for DeepSeek-V4-Flash TP4 at small batches on Blackwell.

An Unexpected Gift for AMD

MI355X users get a thick package this cycle: Kimi K3 gains a grouped-head MLA verify kernel with 1.37-1.77x throughput; GLM-5.2 on gfx950 gets fp8 MLA absorbed bmm at +14% throughput and -12.5% TPOT, with a dense short-context fallback cutting TTFT another 22-43%. The most interesting piece is online NVFP4-to-MXFP4 requantization: at load time, NVIDIA-format weights are converted directly to AMD-runnable MXFP4 without ever holding a full-precision copy, retaining 97.5%-100.2% accuracy across GLM-5.1, Kimi-K2.6, Qwen3.5-397B and three others. Running NVIDIA checkpoints on AMD out of the box used to be an afternoon of pain.

Diffusion Models and Upgrade Notes

SGLang-Diffusion extends breakable CUDA graphs to more models: LTX-2 on H200 drops end-to-end from 10.75s to 6.90s, SANA at 1024px gets -26%. Seven new models are supported, including Muse Glimmer, SANA-Video, and LTX-2.5. Two upgrade gotchas: the CUDA stack moves to torch 2.13.0, and every compiled-kernel cache consolidates under SGLANG_CACHE_DIR — the first launch after upgrading recompiles once, so teams with pre-warmed images should migrate directories ahead of time.

So What

New frontier models ship every week, but inference frameworks decide their real cost at deployment. The signal from v0.5.18 is clear: cold start, communication primitives, and cross-hardware portability are now the main battleground between vLLM and SGLang. Next time your service scales out too slowly, check whether the framework needs an upgrade.

Reference: SGLang v0.5.18 Release Notes (https://github.com/sgl-project/sglang/releases/tag/v0.5.18)