arXiv 2606.20474v1 introduces UltraQuant, a 4-bit KV cache quantization method that achieves 3.47× TTFT (Time To First Token) speedup for Agentic long-context inference. The standout: 4-bit quantization with negligible quality loss, even on 1M-context Agent workloads.
The technical details: UltraQuant uses a "channel-wise 4-bit quantization" scheme, where each channel of the KV cache is quantized independently with a learned scale and zero-point. The "channel-wise" design is the key — it adapts to the per-channel distribution, avoiding the "outlier channel" problem that plagues uniform 4-bit quantization.
The "1M-context Agent" highlight: UltraQuant is specifically designed for Agent workloads, where the context can grow to 1M+ tokens. At 1M context, the KV cache occupies hundreds of GB even at FP16, and TTFT can exceed 30 seconds. UltraQuant's 4-bit quantization reduces the KV cache to ~50GB, cutting TTFT to under 10 seconds.
The benchmark: on the LongBench-Agent benchmark (1M-context Agent tasks), UltraQuant-4bit hits 3.47× TTFT speedup over FP16, with quality loss of less than 0.5 points on the downstream task. The quality is preserved because the "channel-wise" design adapts to each channel's distribution.
The bigger takeaway: "Agentic inference" has a different optimization profile than "chat inference." Chat inference is dominated by generation speed (tokens/sec), while Agent inference is dominated by TTFT (because the Agent must wait for the full context to be processed). UltraQuant is the first method to specifically target Agentic inference, and the 3.47× speedup is a significant result. For the industry, this signals that "Agentic optimization" is becoming a real sub-field of LLM inference.