When you run a large language model on a consumer GPU, the real ceiling is often not the weights — it is the KV cache.
Weights are loaded once, but the KV cache grows with every decoded token. Datacenter GPUs absorb that cost with large memory and high bandwidth. Swap in a laptop with 8GB of VRAM and the same cost shows up directly as throughput collapse, latency spikes, and OOM failures that arrive long before the model's nominal context window.
A paper posted on arXiv on October 13, 2026 (SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware, https://arxiv.org/abs/2610.18140) takes a shot at this old problem. It does no compression and no quantization. Instead it splits "the cache does not fit" into three runtime decisions that can be made online.
Three decisions rather than one compression ratio
The paper sorts prior work into three disconnected lines: layered-offload systems such as FlexGen and LeoAM answer "can it run at all"; cache-retention methods such as StreamingLLM and H2O answer "which information to keep"; local-recovery methods such as Quest and ProphetKV answer "how to get it back". SparPR's claim is that these three questions belong in a single runtime loop.
Query-Aware key-block selection. Each historical block gets a compact precomputed summary — the mean key vector across attention heads within that block. These summaries stay resident on the GPU, so scoring touches only the summaries and never the full KV blocks offloaded to CPU or disk. A block's relevance score is the sum of inner products between the query and each head's summary. The paper uses Top-k rather than a fixed threshold so that the selection scale stays stable as the candidate block count changes.
Dynamic budget control. The system estimates the copy cost and the recompute cost of a single block separately. Copy cost is computed from data volume divided by host-to-device bandwidth, with a factor of 2 accounting for both key and value; recompute cost scales with block length and the square of the hidden dimension, and on the sparse path the KV states are rebuilt from a checkpoint through the attention projection alone — no MLP rerun, no full forward pass. The ratio of the two costs maps to a target keep ratio, which is then bounded by a Minimum Keep Floor so the budget never collapses too aggressively under pressure.
On-demand local recomputation with cost-aware fallback. The blocks picked by Top-k are usually scattered across the context, and recovering them one by one brings fragmentation overhead. SparPR therefore merges adjacent blocks into continuous spans, then compares the cost of full recovery against sparse recovery. Once the sparse path is no longer clearly cheaper, the system falls back to the dense path — that is what the Dense Fallback Threshold γ governs.
Measured numbers on one laptop
The setup is deliberately concrete: a consumer laptop with an RTX 3070 Ti 8GB, an AMD Ryzen 9 6900HX mobile CPU, 16GB of host memory, Windows with WSL, FlexGen/FlexLLMGen as the execution engine, and facebook/opt-6.7b at batch size 4.
DeepSpeed and HF Accelerate run out of memory before 1k tokens, and a Quest-style query-aware scheme OOMs at 1200 tokens. Only FlexGen and SparPR stay runnable. FlexGen completes 1k through 4k by spilling to CPU and disk but times out at 8k; SparPR cuts end-to-end latency by 31%, 52%, and 63% at 1k, 2k, and 4k tokens, raises throughput by 1.4×, 2.1×, and 2.7×, and finishes an 8k-token workload in 44 minutes.
Fidelity is measured with a needle task: a target fact is inserted at two depth positions in an excerpt of Alice in Wonderland. At a 50% keep ratio SparPR scores .943/.879/.780 at 2k/4k/8k, while H2O scores .492/.506/.511 and StreamingLLM scores .445/.467/.485. At 10% the gap widens sharply: SparPR holds .600/.482/.308, H2O drops to around .12, and StreamingLLM collapses to .02. The paper's explanation: StreamingLLM is position-based, so a needle survives only if it lands in the recent window; H2O is query-agnostic, so its hit rate roughly tracks the keep ratio.
Two ablation results are worth noting. Removing dynamic budget actually raises accuracy (.963 versus .957 for the full version), which suggests the budget mechanism mainly pays off at longer contexts. The Quest-only variant reaches just .777, the lowest of any variant — the latency it saves is bought with retrieval quality. The paper is candid about the limits: a single OPT-6.7B model, one consumer GPU, and one constructed needle task.
So what
The value of this paper may not lie in the headline percentages but in a more pragmatic framing: bottleneck management for on-device inference is fundamentally a running trade-off between moving data and computing it, rather than the search for a one-and-done compression ratio.