Long-context inference is one of the core competitiveness of LLMs, but the memory overhead of KV Cache grows linearly with sequence length, becoming the main bottleneck of inference efficiency. Existing KV Cache pruning methods commonly use cross-layer uniform pruning strategies, implicitly assuming all layers contribute equally to model performance. However, Transformer layers have significant differences in sensitivity to pruning.
Uniform Pruning, an Overly Optimistic Assumption
Most existing methods apply the same pruning ratio to all Transformer layers, implicitly assuming that each layer contributes equally to model output.
The paper DepthKV (Layer-Dependent KV Cache Pruning for Long-Context LLM Inference) published on arXiv on April 27 punctures this assumption. The researchers found that Transformer layers' sensitivity to cache pruning varies dramatically — some layers are very tolerant of token dropping, while others are highly sensitive. Uniform pruning is either too conservative to be effective, or too aggressive to damage model quality.
DepthKV: Allocating KV Budget by Layer Sensitivity
DepthKV's core idea is no longer using a fixed global pruning rate, but two steps:
- Measure each layer's pruning sensitivity: quantify each layer's impact on downstream task performance after dropping different ratios of KV through ablation experiments;
- Allocate budget based on sensitivity: distribute the fixed global KV budget by layer — sensitive layers keep more, insensitive layers drop more.
The experiment covers multiple models and tasks, and under the same global pruning rate, DepthKV consistently outperforms uniform pruning. This shows adaptive inter-layer budget allocation is more effective than simply increasing the pruning rate.
Implications for the Industry
The significance of this work is more than just another optimization paper. Behind it lies a larger trend: as 1-million-token context becomes the standard for top models, KV cache management has shifted from performance tuning to a precondition for being able to run at all.
Google has TurboQuant compressing KV cache to 3-bit (lossless precision), DeepSeek has sparse attention halving long-context inference cost, and now DepthKV provides a new solution at the pruning-allocation layer. Several technical routes are converging — 2026's LLM inference efficiency war has just begun.
For developers, DepthKV's thinking (identify layer sensitivity, allocate resources on demand) can also extend beyond KV cache to more VRAM-bottleneck scenarios like activation cache, attention computation.