Most inference frameworks extend a paper: the numbers work in the paper, then the serving stack needs half a year of tuning. DeepSeek ran it the other way round. On October 20 it open-sourced DeepInfer under the Apache 2.0 license, the engine behind its production services. The repository showed 18,000+ stars and 1,900 forks within two days, while the code itself is four commits deep.

What shipped, and what did not

The README is blunt: DeepInfer is "the high-performance inference framework that powers DeepSeek's large-scale production services", and what ships now is "offline inference with the core features needed for serving at scale". The default checkpoint is DeepSeek-V4.1-Flash-1017, and it is the only supported model.

The feature list runs long: attention data parallelism and balanced context parallelism, MoE expert parallelism with EPLB, DSpark speculative decoding with confidence-scheduled verification, SWA bounded replay, asynchronous scheduling via a zero-overhead batch scheduler, batch-invariant inference, deterministic sampling, and structured outputs. At the kernel layer it calls DeepSeek's own FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect repos, plus custom TileLang kernels.

What is left out is stated just as plainly: the KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas, with the reason given as tight coupling to internal infrastructure. What you get is the engine, not the service. A companion library, deepseek-recipe, converts requests into token sequences using the same logic as DeepSeek's official API.

A very short hardware list

Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, nothing else. SiliconANGLE notes that Hopper and older NVIDIA architectures are absent, so teams holding only A100s or H100s cannot use this. Batch-invariant inference and deterministic sampling are also NVIDIA-only for now.

Two design choices worth reading

SWA bounded replay. Parts of V4.1 Flash use sliding-window attention, and that slice of the KV cache stays in memory. On a cache miss, the old approach recomputed along "number of layers times window length". The new one replays only the final 128 tokens of each prompt, with attention computed inside those 128, pinning the recompute cost so it no longer scales with context length or depth. Citing DeepSeek, SiliconANGLE reports this nearly halves total prefill computation.

Confidence-scheduled verification in DSpark. A lightweight draft module proposes five tokens at a time and the target model verifies them in a batch, correcting the first wrong one. A confidence head predicts how likely each draft is to be accepted, so low-probability drafts are skipped and the compute goes elsewhere. SiliconANGLE reports this cuts draft tokens checked per step by about 30%, with DeepSeek estimating a 4% to 7% throughput gain at its production load.

Discount the numbers

Per a deployment Q&A published with the code, one cluster now serves 155 trillion tokens a day, about 38 USD per GPU per hour at off-peak pricing. Those are DeepSeek's own numbers, and every published benchmark ran on NVIDIA hardware: prefill on 4 SM100 GPUs, decode on 64. The same report says the smallest deployment DeepSeek describes as efficient is 64 GPUs for prefill plus 64 for decode. Ascend support ties back to DeepSeek's September 30 move bringing TileLang to Huawei chips; the repo ships requirements-ascend.txt and asks for CANN 9.2.

So what

For most teams this is not something you deploy on Monday: one model, two hardware families, half the serving stack still inside DeepSeek. The value sits elsewhere — the engineering decisions a frontier lab tuned behind closed doors in production are now readable. Sliding-window replay, confidence-scheduled verification, prefix caching: previously guesswork, now source you can read.

Worth keeping in view: a day before this code appeared, the U.S. Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products, including third-party services built on its technology. Open source and geopolitics landed inside the same 48-hour window.

Sources