arXiv 2606.26650 introduces CAT-Q, an ICML 2026 oral, proposing an extremely low-bit quantization method that pushes LLM weights to 1.58-bit while keeping downstream quality. The most striking result: with just 512 calibration samples, CAT-Q can quantize any model from 1.7B to 235B to 1.58-bit, with a training cost 100,000× lower than full-precision fine-tuning.
The technical path: CAT-Q treats quantization as a "binary-coding" problem — each weight is represented by a binary code (3 values: -1, 0, +1, i.e., 1.58-bit). The challenge is to find the optimal code assignment for each weight group. CAT-Q uses a coordinate-descent approach that iteratively adjusts the code of each weight to minimize the output error of a calibration batch.
The "512 samples" highlight: traditional quantization-aware training (QAT) requires the full training corpus; CAT-Q uses just 512 carefully selected calibration samples and achieves comparable or better quantization quality. The selection of these 512 samples is via a "diversity-maximizing" algorithm — pick samples that maximize the activation-pattern diversity of the model.
The result: a 70B model compressed to 1.58-bit occupies about 14GB (vs 140GB at FP16) — fitting in a single consumer GPU. Quality is preserved at 95-98% of the FP16 baseline across MMLU, HumanEval, and GSM8k.
The bigger signal: CAT-Q is the next step in the "extreme quantization" direction. From 8-bit to 4-bit to 2-bit, and now 1.58-bit, the field is rapidly approaching the "weights-as-binary" theoretical limit. The 100,000× training-cost reduction is the key — it makes "any model can be 1.58-bit" a practical reality.