Running an 80-billion-parameter model — a 48.4 GB file — on a laptop with 32 GB of RAM, at 37 tokens per second, is the headline number an open-source project called Janas published this week.

Turning "won't fit" into "reads fast enough"

Janas-LLM is an inference engine written in pure C with no dependencies beyond the C library, released under GPL-3.0. Its core idea is not to shrink the model but to change the question: all weights stay on disk, and during inference each token streams only the few MoE experts its router activates from an NVMe SSD. Experts that keep being hit stay in a memory cache; cold ones are read and released. RAM only has to hold the active working set, not the whole model.

The author, Maurizio "camauri" Cammalleri, built and measured everything on one laptop: Intel Core Ultra 9 185H, 32 GB DDR5, a 1 TB NVMe SSD reading at about 5.7 GB/s, and the integrated Arc GPU through Vulkan. Running Qwen3-Next-80B-A3B, the hot-expert cache takes about 19 GB; the rest is read from disk on demand.

The numbers: 37 vs 2.8, and 13 straight wins

In same-machine comparisons measured on 9-12 October 2026, Janas-LLM read a 1,024-token prompt at 62 tok/s against llama.cpp's 16, wrote at 24 tok/s against 2.8, and reached 37 tok/s in chat with speculative decoding. The author reports that on his machine Janas-LLM was faster than llama.cpp on all 13 models compared in Q4_K_M — spanning dense and MoE builds of the Qwen3, Qwen3.5, Qwen3.6, Qwen3-Next and Gemma 4 families — including small ones like Qwen3.5-0.8B (109 vs 32 tok/s writing).

The 80B case is the sharpest contrast: llama.cpp's usual build ran out of memory as soon as the GPU took part — the CPU backend copies weights to repack them, and the Vulkan backend leaves buffers the system cannot reclaim. Built without repacking and reading through mmap, it managed 2.8 tok/s writing. Janas needed no options at all.

There are honest exceptions. unsloth's 3-bit Qwen3-30B reads a prompt slower than llama.cpp (100 vs 112 tok/s) because those weights do not yet take the GPU fast path — the README says so plainly. Speculative decoding has boundaries too: drafts survived 74-89% on the larger Qwen models, but on Italian prose Qwen3.5-2B kept only about a third and gained almost nothing.

A one-person project, and "this is an experiment"

The README opens with cold water: "This is an experiment, not a product." The public API and command line change several times a week; the engine deliberately fills free memory and reads the disk flat-out with O_DIRECT, which can push a tight machine into swap, and one GPU session triggered a driver reset that took the desktop down. Everything was developed and measured on that single laptop; other CPUs, discrete GPUs and slower disks are untested.

Yet it demonstrates a real route for local inference in the MoE era: sparse activation means model size no longer equals memory need — the bottleneck moves from capacity to bandwidth and cache hit rate. If routing keeps visiting the hot experts, speed holds; once it wanders, the disk becomes the ceiling. It is a different cut of the same direction as Colibri streaming a 552B model from SSD and Berkeley's KV cache offload.

For most users the advice is simple: don't put this in production yet. For anyone watching local inference, the repo — a few hundred stars as of today — is worth following. One person, pure C, zero dependencies, at this level of finish, is itself a counter-argument to "big local models need big VRAM."

Reference: https://github.com/prabanta-dev/janas