arXiv 2606.18114 introduces a 1.58-bit quantization method for Mamba-2 (a state-space model architecture). The result: a 1.58-bit Mamba-2 model that occupies only 744MB — small enough to run on a smartphone, with quality preserved at 95% of the FP16 baseline.
The "Mamba can't be 1.58-bit" assumption: quantization to 1.58-bit is well-established for Transformer models (e.g., the CAT-Q paper). But Mamba-2 was thought to be harder to quantize, because its SSM (state-space model) architecture uses a recurrent state that is sensitive to numerical precision. This paper shows that the assumption is wrong.
The technical details: the authors use a "quantization-aware training" (QAT) approach, starting from a pretrained Mamba-2 checkpoint. The QAT uses a "state-aware" loss that explicitly accounts for the recurrent state's sensitivity to quantization. The result is a 1.58-bit model that is within 0.5 nats perplexity of the FP16 baseline on WikiText.
The "744MB" highlight: a 1.58-bit Mamba-2-1.4B model occupies 744MB, small enough to run on a smartphone with 2GB of free RAM. The inference speed is 50 tokens/sec on a Pixel 8, fast enough for interactive chat. This is a significant result for "on-device LLM" — Mamba-2's SSM architecture is more efficient than Transformer at small scales, and 1.58-bit quantization makes it even more so.
The bigger takeaway: "1.58-bit for non-Transformer architectures" is a significant expansion. The CAT-Q paper focused on Transformer models, and this paper shows that the technique generalizes to SSM. For the industry, this means "1.58-bit on-device LLM" is now viable for multiple architectures, and the "on-device LLM" market will see significant growth.