Long-context inference rarely dies from compute; it dies from the KV cache. The cache grows linearly with sequence length and eats memory first, on serving GPUs and on edge devices alike. The standard fix is eviction: score every cached token, and when the cache exceeds a budget, drop the lowest-scoring ones. A paper posted on arXiv on September 20 by Qualcomm AI Research — ValueDiff — starts from an awkward observation: the foundation most eviction methods stand on is being dismantled by model architectures themselves.
The anchor old methods rely on is disappearing
Most eviction policies bet on attention sinks: a few tokens absorb disproportionate attention mass and anchor the context. StreamingLLM preserves sink tokens outright. H2O, TOVA and SnapKV score by attention. KeyDiff and ManifoldKV lean on key-vector geometry. All of these signals live on the key side. But QK-normalization, gated-attention hybrids, learned attention sinks and logit softcapping — the machinery behind recent models — all suppress sinks. The paper measures it: Llama 3 shows sink rates of 0.86–0.90, while Qwen3.5 and GPT-OSS-20B fall to nearly zero. When the anchor dissolves, anchor-based methods misfire.
ValueDiff: score by value-vector dispersion
The paper's empirical hook: the weaker the sinks, the higher the dispersion of value vectors relative to key vectors. ValueDiff's scoring rule is one line — compute each token's L2 distance from the cache mean in value space. Tokens near the mean are near-redundant and evicted first; tokens far from it carry distinctive content and stay. There is also a theoretical footnote: under a max-entropy assumption about future attention, this is provably the eviction that least disturbs the attention output. No attention scores involved; the signal is query-invariant.
The numbers
On RULER at a tight 2k-token budget, ValueDiff retains 88–99% of dense attention performance across seven sink-suppressed models, ranking first on six of seven. On LongBench at the 4k budget it averages 92% retention versus 83% for the strongest prior baseline. The collapse cases are more striking: on Gemma3-4B, KeyNorm retains only 49.5%; on GPT-OSS-20B, KeyNorm drops to 25.3% and ManifoldKV to 31.0% — effectively unusable. It holds in reasoning too: on MATH-500 at a 25% cache budget, ValueDiff is the strongest non-dense method on every sink-suppressed model tested, with attention- and key-based alternatives trailing by roughly 19–24 points on Qwen3.5. The memory math is concrete: Qwen3.5-4B at 128k context sees peak GPU memory fall from 15.0 GB to 8.6 GB (−43%); GPT-OSS-20B at 65k saves 29%.
So what
The lasting lesson is not another eviction algorithm — it is the methodology. Inference research has to recalibrate against architectural drift: training-side changes made for stability and long context (QK-norm, gating) hand the bill to cache policy at deployment. The authors themselves flag hybrid key/value scoring as future work, which says no single signal is the endgame. For anyone deploying models: before picking a KV eviction policy, check your model's sink behavior first — on new architectures, the old defaults may not be merely suboptimal; they may collapse.
Reference: arXiv:2609.23314