A 313B-parameter model in fp8 occupies 328GB of disk, while a 16GB laptop cannot even hold a fraction of it — the first wall every "run big models locally"方案 runs into. The usual answers are brute-force quantization, distilled small models, or just giving up and calling the cloud. WARP (formerly WASTE), an open-source engine from SQLite Cloud, offers a fourth path: let the weights live mostly on NVMe and keep RAM as a cache.

Streaming experts from NVMe instead of stuffing them into RAM

WARP is an embeddable inference engine written in pure C, with the inference path depending only on libc and pthreads, under Apache 2.0, and already at 2.3k stars on GitHub. The design is tailored to MoE architectures: the shared trunk stays resident in memory, the experts actually activated by each token are streamed from disk on demand, and all remaining RAM serves as a bounded expert cache. The container format is arranged so that one expert corresponds to exactly one aligned read; a lookahead router predicts the experts needed by the next layer and starts reading early, overlapping IO with compute.

Quantization is tiered: experts use 3-bit residual vector quantization while the more sensitive shared weights stay at 4/8 bits. Combined with Kimi K3's linear attention and compressed latent KV cache — about 0.21GB at 4K context instead of 11.25GB — opening the entire K3 requires only 29.19GB of memory.

Measured: the numbers and the failure modes

Measured on a 64GB MacBook Pro (M5 Pro, internal SSD):

  • GLM-5.3-Flash (313.89B total, 17.31B active): runs with as little as 5.14GB of RAM, measured at 3.32 tok/s over 64 tokens and 3.86 tok/s over 200 tokens; a 16GB machine automatically drops to 3.06 tok/s, about 90% of the 64GB figure
  • Kimi K3 (2.78 trillion parameters): the full model, not distilled or pruned, at 0.45–0.62 tok/s; a cold token reads about 17GB of experts
  • Kimi-Linear 48B: 17.22 tok/s with a 1.32GB memory floor

The most honest parts of the README are the negative results: a bigger expert cache is not always better — past the budget the hit rate keeps climbing while throughput collapses eightfold, because a cache hit becomes a page fault; and dropping experts per token from 16 to 8 buys a 1.49x speedup (KL divergence 0.037), while dropping to 4 breaks the model outright. Storage is the main constraint: the internal SSD sustains 12.78GB/s versus 0.94GB/s on the tested USB enclosure.

Details worth remembering

  • The whole project is "human-driven, LLM-written code" — the author says this is the only way to iterate on algorithmic hypotheses fast enough, with the ultimate goal of having K3 improve the engine itself locally
  • All layers are checked against a PyTorch reference; GLM's final logits agree within relative L2 of 2.41e-5, with identical argmax and top-10
  • Since 0.6.8 the converter handles the DeepSeek V3/R1/Kimi K2 family; GLM-5.3-Flash's vision tower (282MB) loads on demand, and a 200×140 image costs only 40 token positions

On the question of whether local hardware can run large MoE models, WARP's answer is to move the bottleneck from RAM to NVMe bandwidth — and every laptop has that. The project was originally named WASTE, with the author's footnote: every cloud token is paid for twice, once on the invoice and once in datacenter electricity. As open weights keep getting more generous and SSDs keep getting faster, "the machine on your desk can actually run this" is turning from consolation into engineering fact. See the GitHub repository.