For the past three years, the compute bill for large models has been dominated by "more parameters = smarter." PrismML, a startup from Caltech, threw a counter-direction bomb on September 17: ternary-compressed Qwen3.8 27B to 5.9 GB, scoring 83.9 on a 20-benchmark aggregate while retaining 98.2% of the original model's capability, running at 143 tokens/s on RTX 5090, free under Apache 2.0. The number mix means an ordinary PC, even a high-end smartphone, can now run intelligence density close to cloud 27B models.
What ternary weights change
Normal weights sit at 16 bits each. Bonsai 2 stores only +1, -1, and 0. Information density collapses from 65,536 states to 3, paying precision for a 9-10x memory compression. 5.9 GB lands squarely within both PC memory budgets and high-end phone flash storage.
Why ternary rather than binary — the value 0 acts as a "silencer" in matrix operations, forcing part of the weight channels to zero; that sparsity becomes a stability source.
98.2% is not marketing
The 20 benchmarks span reasoning, math, coding, instruction following, vision, and agentic tool use — a rare "real-work" test suite, not just MMLU. Bonsai 2 27B averages 83.9, Qwen3.8 27B is 85.4 — a 1.5-point gap, 98.2% retention. Up 3 points from the first Bonsai's 95%, evidence each compression generation converges at visible rate.
This retention ratio is uncommon in low-bit quantization. Most 4-bit schemes drop 2-4 points; Bonsai 2 has reached the "zero-cost" zone.
Where the counter-direction lives
Industry convention pushes parameter scale up — GPT and Claude double total parameters nearly every generation, at the cost of cloud inference costs growing exponentially. Most on-device low-bit schemes serve "people already decided to run large models"; the 5.9 GB size turns "whether to run on-device" from a cloud-locked choice into something a personal device can absorb.
Ion Stoica (Berkeley Sky Computing Lab director, Databricks co-founder) summarized it as "local intelligence, free because the device is already paid for." When a model fits in a user's pocket, the cloud API moat shifts from "model capability" to "toolchain and distribution."
Engineering implications of 5.9 GB
The 143 tokens/s on RTX 5090 means real-time is no longer a bottleneck. With the model fitting within a 24 GB VRAM budget, developers can deploy an entire 27B-class agentic pipeline on a single machine, no cloud API required. Direct upside for teams building local knowledge bases, private document Q&A, and offline coding agents.
PrismML's plan is to apply ternary compression to "several-hundred-billion-parameter" larger models — CEO Babak Hassibi told TechCrunch that the larger the model, the more "retained intelligence" survives compression, because the larger weight space can tolerate precision loss. If this law holds, 100B+ models will fit on a single workstation in the future.
So what
The real signal of Bonsai 2 27B is not "yet another open-source 27B" — it is the rate at which compression technology itself improves. 95% → 98.2% in one generation; continuing that trajectory, 100% retention is not far. When benchmark retention flattens, "on-device LLM" stops being a marketing term and becomes engineering reality.
For developers, the dividend of this ternary compression wave is lowering the inference infrastructure floor. For model vendors, the "sell API calls from bigger models" business model will face its first real competition from the device side. The open-source community's key question for the months ahead: can 100B+ models be compressed at the same retention rate? If yes, the endgame is making the cloud an option.