As large model parameters balloon from billions to trillions, can they run on phones is becoming an increasingly hard question to answer. A joint team from Nanjing University and Microsoft AI recently released the EdgeRazor framework, providing a striking answer — at just 1.58-bit precision, Qwen3-0.6B's storage compresses from 1.41GB to 280MB, decoding speed improves 15×, while maintaining usable inference capability.
The core challenge of this research: existing three mainstream quantization methods each have limitations — post-training quantization degrades sharply below 4-bit, quantization-aware training requires massive compute for gradient updates, and while quantization-aware distillation strikes a balance between the two, it still relies on manual intervention for layer-level feature selection.
EdgeRazor breaks through these bottlenecks with three innovative modules. Mixed-precision quantization-aware distillation allows fine-grained precision allocation across matrix layers, allocating more bits to sensitive layers to preserve key information. Adaptive feature distillation lets the compressed student model intelligently select the most representative layers from the teacher model for supervised learning. Entropy-aware KL divergence further extends the use of distillation data, letting the model maintain stable training on both manually-labeled and model-generated data.
Experimental results show EdgeRazor at 1.88-bit precision already surpasses all 3-bit precision competitors, with performance 11 percentage points higher than the most advanced 2-bit post-training quantization methods across 14 different domain tasks. The practical significance: truly runnable on consumer-grade hardware while maintaining usable LLM capability, edge deployment is no longer a luxury.
The core insight: extreme compression is not a single-tech breakthrough, but a deep fusion of quantization and distillation. Mixed-precision allocation is essentially fine-grained regulation of the information bottleneck for different weight groups — more bits for sensitive layers, decisive drops for redundant ones. This work from Nanjing University and Microsoft AI provides a direction worth tracking for extreme low-bit quantization.