The dominant cost of LLM inference is weight memory. Post-training quantization (PTQ) compresses weights to 4-bit and below without retraining, which makes it the pragmatic choice for most deployment teams. But mainstream methods share an under-discussed weakness: numerical stability -- the problem is not the average score, it is that certain layers suddenly blow up. A paper submitted to arXiv on September 10 (arXiv:2609.11687) seriously engineers a mathematical route dating back to 1977. The authors come from the Institute of Numerical Mathematics and the Steklov Mathematical Institute, and the author list includes Boris Kashin himself -- the 1977 decomposition theorem carries his name.

The crash sites

In the paper's comparison experiments, QuIP-family instability is not incidental. On Pythia-6.9B, QuIP and QuIP-RG diverge beyond 2000 WikiText-2 perplexity, the QuIP# variant with fine-tuning and the vector codebook disabled still blows up to roughly 325, while Kashin-DCT+H lands at 20.6±1.6 -- more than an order of magnitude better than QuIP# at the same bit budget. Mistral-7B v0.1 is sharper still: all four QuIP variants abort with NaN in LDL back-substitution on the SwiGLU mlp.down_proj layer, and OPTQ itself degrades to roughly 380 perplexity; Kashin-DCT stays numerically stable, within about 0.3 of the FP16 baseline of 8.63. Per the paper's own reporting, across all four stress configurations they encountered, this is the only pipeline that stays stable on every layer.

Three surgical changes

The method is a rebuild of Kashin-decomposition quantization with three fixes:

  • Dense orthogonal matrix replaced by a sign-randomized DCT. The original scheme stores an N×N random orthogonal matrix with O(N²) per-iteration cost; a sign-randomized Discrete Cosine Transform cuts this to O(N log N) with zero stored matrix.
  • A partitioned greedy algorithm with alternating updates. Fixing the update order in blocks of four guarantees the four-peak distributions required for stable 2-bit clustering of both factors, with a proven geometric convergence rate -- no longer luck of the draw.
  • Closed-form cluster-center initialization. Peak locations ±c1±c2 are known analytically from residual norms and directly initialize k-means, eliminating the multi-restart search that previously dominated quantization wall-clock time.

Around this core, the pipeline composes OPTQ-style sequential error compensation and QuIP-style incoherence preprocessing (Hadamard or Kronecker), implemented in compile-friendly JAX with multi-GPU support.

Results: stable, without giving up the average

At 4-bit per channel on OPT-1.3B, Llama-2-7B/13B, and Pythia-1.4B, the method is competitive with OPTQ, QuIP, QuIP-RG, and a fine-tuning-free variant of QuIP#. On Llama-2-7B WikiText-2 perplexity: FP16 baseline 9.20, Kashin-DCT+H at 9.60, slightly better than OPTQ's 9.80; on the same configuration QuIP's variance reaches 48.30±29.84. At inference time each weight decomposes into two 2-bit factor codes per channel, structurally suited to native-2-bit hardware. Experiments ran on a single H100, calibrated on 1000 sequences of length 2048 from WikiText-2, evaluated through lm-evaluation-harness.

The paper is upfront about limitations: no comparison against AWQ, OmniQuant, or rotation-based activation-quantization baselines; it only argues that rotation preprocessing composes orthogonally with this factorization.

So what

The notable part of this paper is not benchmark chasing -- it does none -- but elevating "worst case does not explode" to the same rank as "average perplexity". What kills deployments is never a 0.2 perplexity gap; it is one layer silently returning NaN and the whole service going down. When native 2-bit hardware actually arrives, the methods that survive first are the ones that get to talk about accuracy at all.