arXiv 2606.20381 introduces UFP4 (Unbiased FP4), a method from Ant Group's Bailing team that addresses the "shrinkage bias" problem in E2M1 FP4 training. The result: stable FP4 training with quality matching FP8, enabling a 2× throughput improvement over FP8.
The FP4 problem: FP4 (specifically the E2M1 format) is the next step in low-precision training, promising 2× throughput over FP8 and 4× over FP16. But E2M1 FP4 has a "shrinkage bias" — the small number of representable values (only 8 positive values) introduces a systematic bias that causes training to diverge or plateau.
The UFP4 fix: a "bias correction" technique that estimates the shrinkage bias per layer and compensates for it. The estimation is done via a running statistic (similar to batch normalization), and the compensation is applied as a per-tensor scale factor. The technique adds no extra compute and no extra memory.
The benchmark: UFP4-trained models match the quality of FP8-trained models on MMLU, HumanEval, and GSM8k, while running 2× faster on H100. The training is stable — no divergence, no plateau, and the same hyperparameters as FP8 training can be used.
The bigger takeaway: "FP4 training" is finally practical. The "shrinkage bias" has been a major blocker, and UFP4's bias correction is a clean, general solution. For the industry, this means the next generation of foundation models will be trained in FP4, with 2× the throughput of FP8. The "compute cost of frontier model training" will drop by another 2× in the next 12-18 months.