Local inference has a long-standing problem: open-weight model sizes keep climbing while consumer memory does not, and typical desktops have few good options for 75B-class MoE models. llama.cpp v0.4.0, released September 4, targets exactly this bottleneck (GitHub release).

On-demand tensor reading: --lazy-mode changes how models load

The headline change is lazy tensor reading (PR #27794): tensors are no longer loaded into memory all at once. The runtime reads each tensor from disk only when inference actually touches it, toggled by the --lazy-mode flag (#27969). This ships with a set of memory-governance changes: preventing RAM peaks during model load (#27483), a new max_buf_size cap for the quantizer (#27795), and row-slab streaming during quantization (#27830). For large MoE models, the combination aims squarely at the "small memory, big weights" scenario.

Another notable option is --n-cpu-ffn (#26622), paired with per-layer expert routing/FFN (#28323): the feed-forward portion of a MoE can now be assigned to CPU or GPU layer by layer — hybrid deployment granularity moves from "the whole model" down to "each layer".

New architectures: Qwen3.8-Flash-Next and the 75B Nemotron-3-Puzzle

Four new names appear in the model list: Qwen3.8-Flash-Next (via the new qwen4exp architecture, #27742), NVIDIA's Nemotron-3-Puzzle-75B-A9B (#25444), DSpark support for Nemotron 3.5 (#27804), and nanbeige4.2-3B (#27730). One caveat worth stating plainly: Qwen3.8-Flash-Next is initial support — the release notes explicitly say optimization improvements are still pending, and a string of qwen4exp fixes followed (#27941, #28123, #28023).

On the multimodal side, DeepSeek-V4 vision input handling was fixed (#28154), DeepSeek-V4-Flash-Vision-Exp support was added (#28133), and video input parameters --video-* landed in mainline (#24318).

ggml 0.23.0 underneath: sparse attention and RDMA

Below the engine, ggml moved from 0.22.0 to 0.23.0. Two additions stand out: sparse flash attention (the new operator ggml_flash_attn_ext_set_n_kv_max), already applied to DeepSeek-V4/GLM and qwen4exp (#27970); and Apple RDMA as an RPC transport (#26421), which — together with RPC event and async backend APIs (#18626) — keeps widening the path for spreading a large model across multiple machines.

The server picked up practical updates too: per-slot context limits (#24124) let concurrent requests sharing one server stop squeezing each other's context, and preserve_reasoning is now enabled by default (#28174).

So what

This repository sits at 127k stars — evidence that "running models on your own machine" is a real, widespread need. And the most common hardware constraint is precisely not compute, but memory that cannot hold the weights. Lazy loading, the quantizer memory cap, and per-layer routing in v0.4.0 all strike at this exact point. If your machine is memory-tight but you want to run the new open MoE models, this version is worth upgrading for; users coming specifically for Qwen3.8-Flash-Next, however, should wait a bit — between initial support and well-optimized, there are usually a few minor versions of distance.