Background: NVFP4 pushes "low-bit" from research to production

Over the past year or two, LLM inference cost has been pushed down step by step. After INT4/INT8, the industry has started looking at more aggressive 4-bit floating point. NVFP4 is NVIDIA's hardware-native FP4 microscaling format — every contiguous 16 elements share one FP8 scale, plus a tensor-level FP32 scale — which gives finer expressiveness than INT4 and saves another band of bandwidth vs FP16. The problem: in real LLM activations, a small number of channels can have values several orders of magnitude larger than the median. At quantization time, those outliers directly set the block scale, forcing the "collateral" ordinary values in the same block into the coarse grid of FP4 — and that is where the accuracy is lost.

A team from South China University of Technology and Intellifusion (arXiv 2609.00066, EMNLP 2026 main conference) gave this problem a name: Collateral Quantization Error, and proposed a PTQ scheme that does not need extra operators and barely adds any inference overhead: OCGQuant (Outlier-Companion Grouping). The code is open-sourced at https://github.com/Eshamont/OCGQuant, and the paper is on arXiv:2609.00066.

Core idea: replace the "collateral damage" with quiet channels

Conventional methods either go mixed-precision (LLM.int8, Atom, using INT8 to catch outliers), Hadamard rotation (QuaRot, etc.) to spread outlier energy, or augmented residual channels for residual compensation (ARCQuant). These work for INT quantization, but do not transfer cleanly to NVFP4 for two reasons: the FP4-E2M1 non-uniform grid cannot be modeled by a single uniform step, and the block-level scale only covers 16 channels, which means the channel order directly determines block composition.

OCGQuant's lever is the latter: since the block is sliced by channel order, reorder the channels. The procedure:

  • Run one forward pass on a calibration set, record each input channel's RMS;
  • Sort by RMS descending; pair each high-RMS outlier channel with the 15 lowest-RMS "quiet" channels from the unassigned pool;
  • Apply the same channel permutation to the weights, and use GPTQ offline to fix the weight-side quantization error in the reordered space.

The intuition is plain: low-RMS channels are already close to 0, and FP4-E2M1's grid is denser in the small-magnitude region (the authors' Lemma 1: dFP4(z)² ≤ ½z² + 1/32), so under the outlier-enlarged scale, the loss when quantizing "quiet neighbors" is bounded.

The paper also gives a formal proof for "why pick low RMS as companions": when the outlier candidate is fixed, the only companion-dependent term in the bad-event upper bound is ∑rf², so picking the G−1 smallest-RMS channels is indeed optimal.

Numbers: Qwen3-1.7B nearly ties FP16

The experiments run on Llama3.1-8B, Llama3.2-3B, Qwen3-8B, Qwen3-1.7B, with perplexity measured on WikiText-2 and downstream accuracy averaged across six zero-shot tasks (excerpted from Table 7):

Model FP16 NVFP4 RTN OCGQuant
Llama3.2-3B 7.81 8.62 8.37
Llama3.1-8B 6.24 6.95 6.75
Qwen3-1.7B 16.67 19.25 16.71
Qwen3-8B 9.72 10.07 9.90

The brightest result is Qwen3-1.7B: RTN inflates PPL from 16.67 to 19.25, OCGQuant pulls it back to 16.71, essentially tied with FP16. Compared with the same-generation NVFP4 method ARCQuant, OCGQuant is lower on all four models (ARCQuant: Llama3.1-8B 6.87, Qwen3-8B 10.35). The Appendix also re-runs RPTQ-style (channel clustering ported from INT to NVFP4), which actually degrades below RTN on Qwen3-1.7B — proof that reordering rules designed for INT uniform grids have side effects on non-uniform FP4.

Inference cost: prefill close to RTN, memory matches RTN

  • Prefill throughput (Llama3.1-8B, seq=2048, batch=16): FP16 11556 tokens/s, RTN 26838, OCGQuant 26439, close to RTN; ARCQuant only 22784. The paper reports "OCGQuant achieves up to 2.29× prefill speedup" relative to FP16.
  • Decoding memory (same model, batch=1): FP16 15.33 GB, RTN 6.02 GB, OCGQuant 6.02 GB; ARCQuant 6.86 GB, the extra is its residual channels.

In other words, ARCQuant trades "an extra residual path" for accuracy, while OCGQuant trades "a channel reorder" for accuracy. The latter adds nothing new to the forward path, so memory tracks RTN, prefill speed tracks RTN — and on production deployments, those two metrics are the bill.

Offline cost and reproducibility

  • RMS profiling: 43.52 s (Llama3.1-8B) / 45.04 s (Qwen3-8B)
  • Companion selection: 0.55 s / 0.89 s
  • Reorder: 1.97 s / 4.52 s
  • GPTQ reconstruction: 248.55 s / 250.44 s
  • Total: ~5 minutes per 8B model

Robustness across calibration sets and sample sizes is also good: across three calibration sets (WikiText-2 / C4 / Pile) and four sample sizes (64/128/256/512), the standard deviation of six-task average accuracy on Qwen3-8B is around 0.10, max 0.24. 128 calibration samples is enough; going larger does not buy more.

A few notes

NVFP4 is on the radar because post-Blackwell hardware turns 4-bit microscaling into a native operator — that is already happening, which is why NVFP4 PTQ methods have been popping up densely in the past six months. OCGQuant's value is not "another SOTA", it is "an engineering path that barely adds inference overhead": the reordering is computed offline, the channel permutation at runtime is fixed (the paper ships a fused CUDA kernel + contiguous reorder-segment fast path), no new operators, no mixed precision, no new hardware.

Two follow-ups worth watching: (1) the author team's Intellifusion + SCUT combination means Chinese teams are advancing in lockstep with the NVIDIA NVFP4 roadmap; (2) the channel-locality + cross-token persistence finding (Figure 8) is hard evidence — outlier channels are "narrow bands stuck to channels", which means future NVFP4 outlier-aware PTQ does not need per-token dynamic adjustment; calibrate once, use long-term.

One-line takeaway: NVFP4's "collateral error" is cured by a permutation table, the next question is whether this channel-grouping idea can follow NVFP4 onto NPUs/TPUs when 8-bit is no longer acceptable and 4-bit FP becomes the floor for edge inference.

References: