Low-bit quantization breaks attention first, not weights. HyQuant, a Shanghai Jiao Tong University paper accepted to EMNLP 2026 Main, pinpoints why: attention maps across Qwen3, Llama-3, Gemma 4 and Qwen3.5 show persistent "vertical lines" — a small set of key positions that nearly every query attends to. Quantizing those positions amplifies error across the whole sequence.

Vertical lines: the error is not uniform

The paper's authors added a key set of numbers in the Hugging Face paper-page comment section: the top 5% of key positions plus a 128-token local sliding window capture 82-86% of total attention mass. Identifying those vertical-line tokens costs only 3-5% of runtime. This explains why low-bit KV-cache quantization tends to hurt long-context reasoning — error concentrates on a few heavily attended tokens instead of spreading evenly, and smoothing-based outlier handling cannot remove structurally concentrated error.

Hybrid precision: keep the critical few in FP16, quantize the rest to 4-bit

HyQuant's design follows that structure directly. Vertical-line tokens and the local window stay in FP16; the remaining attention states are quantized to 4-bit K4V4. In the prefill stage it uses a hybrid-precision quantized attention operator. In the decode stage the same principle applies to KV-cache compression, fusing KV dequantization with attention computation so the KV cache is never materialized — saving memory and bandwidth. The authors describe the design as extremely simple, maintaining near-lossless accuracy across diverse tasks, models and datasets.

Reported numbers and their limits

Self-reported single-H100 results from the paper:

  • LongBench average 45.04 versus 44.59 for full-precision FlashAttention-2 (Qwen3-8B, thinking mode); KIVI, SageAttention and KVTuner land at 37.7-40.5
  • Decode kernel up to 3.58x faster than FlashAttention-2 at 32K context; end-to-end decode 1.04-1.17x faster, while KIVI and KVTuner end up slower than FA2 (0.69-0.80x)
  • At batch 16 with a 32K prefix, it is the only method that still runs (231.6 tok/s) — FA2, KIVI and KVTuner all run out of memory
  • Results hold on Qwen3-32B, Llama-3.1-8B and GLM-4-9B

Caveats in plain terms: all numbers above come from the paper and its authors; the code is open-sourced on GitHub, and third-party replication is still pending.

So what

"How many bits" is the old quantization question; "where the error concentrates" is the one HyQuant asks instead. Attention has natural structural sparsity — vertical lines plus a local window — which means precision budget should follow attention mass, not be spread evenly across bits. For teams cutting long-context inference cost, this is a more direct lever than whole-model quantization: attention and KV-cache dominate long-context memory, and the batch-16 survival test shows it addresses a real capacity bottleneck rather than paper speed. EMNLP 2026 Main acceptance means the route has cleared peer review's first gate; the next thing to watch is community replication across more model families.

Reference: arXiv:2608.27875 (https://arxiv.org/abs/2608.27875); code: github.com/jerrysfls/HyQuant