How badly does Qwen3-4B-Thinking fare when squeezed to 4-bit? At BF16 it scores 0.688 pass@1 on AIME25; under the A4W4 setting with the classic rotation-based method QuaRot, the score collapses to 0.004 — effectively zero. Large reasoning models live on long chains of thought: quantization approximation errors accumulate step by step through autoregressive decoding, and a small error early on can wreck the entire reasoning chain. Low-bit quantization for reasoning models has always been harder than for ordinary models — and hardware keeps pushing everyone toward lower precision: Hopper and Ada GPUs support FP8 arithmetic, while Blackwell extends it to FP4, so the tension will only sharpen.
The diagnosis: the first 30 decoding steps are the wildest
The authors of arXiv 2611.09868 made a deceptively simple observation: on reasoning benchmarks such as GSM8K and AIME25, when the model generates a thinking trace, the average activation standard deviation over the first 30 decoding steps is about 20% higher than that of non-thinking decoding. Channel-wise outliers at step 1 run from shallow layers all the way through to deeper ones, and by step 400 the outlier pattern is noticeably weaker. Conventional calibration pipelines feed on generic text and never cover this "violent opening" distribution — a calibration set that diverges from the deployment distribution is the first root cause of reasoning-model quantization collapse.
The fix: four components plus a lookup kernel
CodeQuant2 is a clustering-based LUT quantization framework in which four components each own one piece of the problem:
- Rotation + scaling for outlier suppression: a shared orthogonal rotation matrix paired with layer-wise diagonal scaling, optimized through the Cayley transform to preserve orthogonality, applied only to the query, key, value, gate, and up projections;
- Adaptive Cluster Allocation (ACA): candidate cluster counts K∈{8,16,32}; a first-order importance score decides which matrix deserves extra capacity, while profiled LUT latency serves as the budget constraint — sensitive matrices get 32 clusters, insensitive ones drop to 8, keeping the average assignment-index width at 4 bits;
- Learnable clustering: after K-means initialization the hard assignments are frozen, soft assignments are learned via LoRA-style low-rank updates, and a temperature schedule anneals them back to hard assignments;
- Reasoning-aware calibration (RAC): let the full-precision model first generate 256 tokens on the GSM8K training split, mix with WikiText2 at 9:1, and truncate or pad each calibration sequence to 512 tokens, so the calibration set genuinely contains "the activations of a reasoning opening."
On the inference side, decode-time matrix multiplication is recast as table lookup: products between 4-bit activation levels and weight centroids are precomputed into a lookup table that lives in each SM's shared memory; activation min/max computation, quantization, and bit packing are fused into a single read, and tile sizes are retuned for the limited batch-level parallelism of decode-time GEMM.
The results: collapsed items come back, and it is faster
On Qwen3-4B-Thinking under A4W4, the average score over six task benchmarks rises from CodeQuant's 0.664 to 0.770 (BF16 is 0.838) — a 10.6-point gain in the paper's accounting. The most collapsed item, AIME25 pass@1, climbs from 0.392 back to 0.563; pass@8 goes from 0.500 to 0.800; LiveCodeBench beats CodeQuant by 15 points. The ablation is clean: the four components contribute +4.00, +3.64, +1.94, and +1.08 respectively — none of them is filler. On the systems side, across A6000, L40s, and A100 GPUs, CodeQuant2 reaches an average 2.42x speedup over BF16, peaking at 2.85x, and still retains at least a 2.28x speedup at batch size 16 on A6000. The whole pipeline fits on a single NVIDIA H100, with the three fine-tuning stages taking roughly 30, 15, and 30 minutes. Evaluation was run with EvalScope, four-shot for GSM8K and zero-shot elsewhere, with per-channel weight and per-token activation granularity kept identical across all compared methods.
The gains are not uniform, though: on Qwen3-8B and Phi-4-Reasoning the improvements are more modest — the 8B model's AIME25 pass@1 moves only from 0.446 to 0.479. The larger the model and the less extreme the task, the smaller the marginal return of quantization repair; that regularity has not changed.
So what
The most valuable takeaway from this paper is not any single number but a methodological shift: your quantization calibration set should look like your deployment workload. Running reasoning workloads? Calibrate on tokens from reasoning openings. Running ordinary chat? Generic text is enough. The other lesson is ACA's framing — codebook capacity is an accuracy-latency budget problem, not an accuracy-maximization problem: 8 clusters buys 47 token/s, 32 clusters leaves only 33, and adaptive allocation lands at 44 token/s, nearly the speed of the 16-cluster tier with a higher score to show for it. Paying each matrix the right salary beats giving everyone a raise. Paper: https://arxiv.org/abs/2611.09868