KV cache quantization has spent years on smarter preprocessing and lower fixed bit widths — rotation, normalization, finer grouping — while the coding format itself went untouched: every value gets the same number of bits by default. D-Quant, just posted on arXiv by a Tencent team, takes a different route: it brings rANS entropy coding, a staple of video compression for decades, into the attention kernel, letting frequent values take short codewords and rare ones long codewords, with a "drift" mechanism bending variable-length output back into fixed-size containers. The result: near-BF16 quality at 2.26 bits per value, roughly 7x less KV memory at 64K long context, and 563.6 tokens/s throughput on a single H20.

Why Fixed Width Is a Ceiling

The fundamental limit of fixed-width quantization is that b bits can only represent 2^b quantization levels. Push toward 2 bits and the level count collapses to 4, with cliff-edge information loss. D-Quant's observation: after rotation and normalization, KV values approximately follow a normal distribution — most values cluster near the center, with only a small fraction in the tails. Fixed-width coding assigns the same bits to frequent and rare symbols alike, throwing away that distributional structure. Entropy coding naturally exploits non-uniformity: codeword lengths follow symbol probabilities, so far more quantization levels fit under the same average storage budget.

The trouble is engineering. Entropy coding produces variable-length streams, while efficient dequantization and computation in attention kernels rely on regular memory layouts and fixed-stride access — variable-length data is a disaster in parallel kernels. Prior work like CacheGen and SplitZip only dared use entropy coding for KV transfer (decompress first, compute later); nobody put it directly on the compute path.

Drift: Bending Variable Back to Fixed

D-Quant assigns each token's K or V tensor a fixed-size byte container and entropy-codes the quantized symbols into it with rANS. When a stream exceeds the budget, a small fraction of symbols drift from high-cost symbols to nearby lower-cost ones until the length constraint holds. The paper measures that drift changes only 0.75% of key symbols and 5.00% of value symbols, and no changed symbol moves by more than one reconstruction level — sparse local corrections, not a global precision cut.

The design also unties two old knots. First, quantization level count is decoupled from bit width: at a 2.26-bit budget, keys use 8 levels and values 6 — unthinkable under the fixed-width paradigm. Second, no calibration: preprocessed KV distributions are close enough to a standard normal that the probability model derives analytically, no calibration dataset needed.

The Numbers

On Qwen3-8B, D-Quant scores 86.50 RULER average (BF16 reference: 87.31); on Llama-3.1-8B, 88.94 (reference: 89.69). At 128K context, D-Quant reaches 70.92 / 74.65 while the strongest baselines manage only 57.14 / 66.15. On efficiency: with an 8K prompt, BF16 runs out of memory at batch 64 while D-Quant scales to batch 128 with just 49.4 GiB peak memory; at respective maximum batches, throughput is 563.6 versus 159.5 tokens/s, a 3.5x gain. Push the budget down to 1.96 bits and Qwen3-8B still holds 85.21, again beating every baseline.

Worth noting is the other side of the trade: entropy decoding adds computation, so D-Quant is slower than BF16 at small batch sizes — what it buys is serving capacity under memory-constrained scenarios, not single-request latency. That trade lands exactly right for long-context agent workloads with high concurrency and long sessions.

So What

The KIVI / KVQuant / QuaRot line kept optimizing how to handle outliers; D-Quant shifted the axis of attack: how much water is still left in the coding format itself. For inference-serving teams, 7x KV memory compression plus 3.5x throughput is worth an immediate integration-cost assessment. For researchers, once entropy coding is inside the attention kernel, the mature rate-distortion toolbox from video compression can now be carried over to KV cache. Paper: arxiv.org/abs/2609.19880.