An RTX 5070 with 12GB of VRAM and 64GB of RAM can now run a 125-billion-parameter open-weight model at 94 tokens per second. That is not a vendor demo — it is the self-measured number in the README of Strata, an open-source project whose first commit landed on September 24 and which shipped v0.1.39 on October 4: 845 commits and 10.9k stars in two weeks, all aimed at one thing — moving Qwen3.8-Flash-Next from the server room onto an ordinary gaming PC, with model files pulled automatically from Hugging Face and nothing leaving the machine.
Requirements and speed: a gaming-PC spec sheet
Per the project README, the GPU can be any mainstream NVIDIA RTX 20/30/40/50 card or an AMD RX 7900/9070-class card with 12GB of VRAM or more, alongside 32GB of RAM (64GB fits every size) and about 80GB of disk. The model download is roughly 70GB; at startup Strata loads 35-55GB into RAM and locks part of it for the card — the docs warn the first start may make the PC unresponsive for one to three minutes.
Speed was measured by the authors on two ordinary gaming PCs. On an RTX 5070 (12GB), the most aggressive Q2_0 quant writes at 94 tok/s and reads a 32K document at 2,650 tok/s; the higher-quality IQ3_S size runs at 53 tok/s. On an AMD RX 9070 XT (16GB), Q2_0 writes at 60 tok/s. The README says a 24GB RTX 3090 should reach 100-140 tok/s — note that this is a projection, not a measurement. Two conveniences stand out for regular users: an AI assistant (Claude Code, Cursor, Codex) can perform the whole install by following docs/AI_SETUP.md in the repo, and the engine exposes OpenAI- and Anthropic-compatible APIs on localhost 127.0.0.1:8080 so existing apps and coding agents connect directly.
v0.1.39: decode 6% faster, output byte-for-byte identical
The engineering taste in this release deserves its own note. After reworking the decode path (sub-warp expert packing, batching the PLE and MTP steps), Q2_0 on the RTX 5070 went from 68.4 to 72.7 tok/s (about +6%), IQ3_XXS gained 2.5-6%, and decode after 4K and 32K prompts is another 5-7% faster — yet the project ran ten interleaved comparison runs on all four quant sizes and found the output byte-identical to 0.1.38. Faster without changing a single answer is rare in local inference, where "close enough" quantization is the norm.
The long-prompt path gains more: up to 18.5% on 32K prompts with IQ3_XXS (with a 1,500-slot expert cache), and +26% on a 243K-token input on an RTX 3060. Multi-GPU now has a real story: dual RTX 4090s gain +63% on mixed text and +132% on code (numbers measured by contributors, not the author, who owns one GPU), and a four-way layer split across 16GB cards served 8 concurrent requests at 360 tok/s total versus 120 for one. The release also adds parallel request slots, experimental engines for older Pascal/Volta and Intel Arc hardware, and an OpenAI Responses API — tested with Codex CLI 0.160.0, with later turns reusing about 96% of the prompt from cache.
If it does not fit in VRAM, stream the experts from RAM and SSD
The idea is not mysterious: Qwen3.8-Flash-Next is a MoE model, so each token activates only a small slice of the experts. Strata spreads the experts that do not fit in VRAM across RAM and SSD and streams them to the GPU on demand — the "streamed expert ring" and --prefill auto in the release notes are exactly this. Quant sizes open up the trade-offs: the Coder build removes half the experts, needs only 32GB of RAM, and reaches 91% of the full model's SWE-bench Verified score by the authors' own measurement; the Swift 1.5 fine-tune thinks for less time before answering; Unsloth's ~4-bit UD-IQ4_XS trades a 94GB download for higher quality.
Some cold water
First, 94 tok/s is the most aggressive Q2_0 size — the quality-oriented IQ3_S drops to 53. Second, the authors explicitly note the Coder build is weaker outside code, including Chinese and other CJK text; Chinese-speaking users should pick a size that keeps every expert. Third, the Pascal/Volta, Intel Arc and older AMD paths are experimental features self-measured by community contributors on hardware the author does not own. Fourth, the 3090's 100-140 tok/s is a projection. In other words, "a 125B model on a gaming PC" holds, but every speed grade needs to be read against your own VRAM and RAM.
Even so, the two-week cadence and the discipline of "faster only if byte-identical" set a high engineering bar for local inference. While cloud API pricing and quotas keep swinging, a gaming PC plus 80GB of disk is your fallback — and that road now genuinely works.