ResKV Stops Treating KV-Cache Compression as Token Deletion: A Residual Ledger for Omitted Context

Long-context inference carries an unavoidable cost. For every newly generated token, a model attends to the Keys and Values of earlier tokens. As the context grows, the KV cache consumes increasing memory and memory bandwidth. Most compression methods therefore score historical tokens and retain only a subset, while others merge evicted states into retained entries. Hard eviction is efficient, but it also removes the evicted tokens’ aggregate contribution to attention. Merging preserves more information, yet it can perturb Keys and Values that should have remained exact.

ResKV, submitted to arXiv on July 31, reframes the problem. Individual evicted tokens do not have to remain in the cache, but their aggregate contribution to the numerator and denominator of softmax attention can be represented as residual statistics. Under the same fixed number of KV slots, ResKV divides the cache into two parts:

• an exact main cache that stores high-priority tokens without changing their Keys or Values;
• a compact residual cache that summarizes the aggregate contribution of omitted tokens.

The residual is not applied as a correction after attention has already been computed. Main-cache tokens and residual entries participate in the same softmax normalization, allowing residual entries to restore both the value-weighted numerator and the probability mass in the denominator. The method neither folds omitted information into retained tokens nor increases the total KV-slot budget. Read the paper.

Two controls keep the residual from overwhelming exact retrieval

Residual information is not equally useful in every layer or KV head. During cache construction, ResKV evaluates several candidate residual-budget ratios with fitting and validation queries. A layer and KV head receive residual slots only when the residual cache reduces attention-output reconstruction error relative to a full cache. Otherwise, the entire budget remains assigned to the exact main cache.

A second control operates during decoding. ResKV measures attention sharpness using the largest attention weight over the main cache. If a query already forms a sharp retrieval peak over exact entries, the dynamic gate down-weights the residual logits to avoid diluting that peak. If main-cache attention is diffuse, residual entries retain more mass and can restore evidence distributed across the omitted context.

Gains are concentrated under tight budgets and unknown future queries

The experiments use LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct on LongBench and RULER, covering retained KV budgets of 10%, 20%, 30%, and 40%. ResKV is attached to AdaKV and SnapKV while keeping the total number of retained KV slots identical to each baseline.

The paper reports improvements in all 32 displayed LongBench configurations, with an average gain of 1.02 points across the two backbones. At 10% and 20% retained KV, the average gains are 1.43 and 1.17 points. On RULER, ResKV improves 63 of the 64 displayed configurations, with an average gain of 3.38 points.

The more practical result appears in the query-agnostic setting, where the cache must be compressed before a future query is known. ResKV improves that setting by 4.54 points on average, compared with 2.22 points under query-aware construction. This resembles real serving more closely: a system usually cannot wait for the user’s next question before deciding which parts of the existing context to retain.

The ablations make the mechanism more concrete. Under a 10% retained-KV, query-agnostic setup, removing the validation proxy reduces two representative RULER task scores by 4.00 and 7.04 points. Removing the dynamic gate reduces them by 4.80 and 4.48 points. Replacing shared softmax with separate normalization for the main and residual caches lowers RepoBench-P by 3.00 points.

This is not free acceleration, but it is closer to information compression

The efficiency analysis shows that ResKV’s peak allocated memory essentially overlaps with SnapKV, indicating negligible extra memory overhead. It also maintains a stable decode rate through 128K context. The trade-off is explicit: evaluating the residual branch and dynamic gate at every decode step gives ResKV lower throughput than SnapKV. The paper does not present the method as a zero-cost improvement; it exchanges a moderate amount of computation for better preservation of attention information under a fixed memory budget.

ResKV’s central point is broader than one implementation: KV-cache compression should not have only a “keep or delete” switch. When evidence is distributed across a long context, individual tokens may look unimportant even though their aggregate contribution changes the answer. The next useful question is therefore not only which tokens to retain, but how omitted information can be compressed into a structure that still participates in inference.