When large language models run long chains of thought, the binding constraint is usually not compute but memory. The key-value cache grows linearly with sequence length, and a reasoning trace of tens of thousands of tokens can exhaust GPU memory outright. The standard remedy is KV cache compression: score past tokens, keep the important ones, evict the rest. But what should the score be based on? Nearly every existing method leans on the same assumption — that queries from the most recent decoding steps are a reliable proxy for future attention patterns.

A team from Hanyang University and Sungkyunkwan University has now falsified that assumption, in a paper accepted to ICML 2026.

Long reasoning has moments of looking back

The paper starts from a phenomenon the authors call Thought Revisiting Tokens (TRT). Models doing long-horizon reasoning do not move strictly forward: at certain decoding steps they re-attend to distant earlier context — typically the task-solving plan formulated early in the trace — in order to maintain global coherence. The attention targets of these queries sit far behind the current position.

That is precisely where recent-query scoring breaks. RPC and R-KV, both designed for reasoning models, estimate which KV entries matter using only the last few queries. TRT targets are invisible to that window, so by the time the model actually wants to revisit its early plan, those entries have already been evicted as low-scoring. Memory is saved; the reasoning chain is broken.

A geometric analysis surfaces something more useful: TRT queries are not scattered. In embedding space they cluster into a small number of similarity groups — and clusters can be summarized by a few representatives.

Beacon queries anticipate what will be revisited

Hence BeaconKV: maintain a compact set of beacon queries as representatives of each global query cluster, and use them to anticipate which KV pairs will be revisited, without storing the entire query history.

The mechanism has three parts. A small set of beacon queries is sampled from the generated reasoning trace and refreshed on the fly during decoding via Continual Farthest Point Sampling. When the decode-time KV cache exceeds its budget, past entries are scored using attention from these beacon queries together with the most recent queries. Only the highest-scoring entries survive, while a recent sliding window is always preserved. Sampling operates in the pre-RoPE query space to capture geometric diversity.

The method is training-free — a pure inference-time modification. The open-source implementation defaults to 16 beacon queries, 16 recent queries, and a window size of 32.

The numbers: up to 31.7 points of accuracy gap

Experiments span four open-source reasoning models (R1-Distill-Qwen-7B, R1-Distill-Llama-8B, Qwen3-4B, Qwen3-14B) across AIME24, MATH-500, GPQA-Diamond and LiveCodeBench. The paper reports accuracy gains of up to 31.7 percentage points over existing compression methods, peak GPU memory reduced by up to 5.8x under aggressive compression, and throughput improvements of over 4.3x relative to the uncompressed baseline.

The efficiency table shows the trade-off concretely. On a single A100 80GB running Qwen3-4B with 32K generation length, Full KV at batch size 14 delivers 82.3 tokens/s, 77.0 GB peak memory and 54.4 LiveCodeBench accuracy. BeaconKV at a 2K budget and the same batch size gives 356.4 tokens/s, 13.3 GB and 51.1 accuracy — roughly one sixth the memory for 3.3 points of accuracy.

The head-to-head with RPC is more telling. At a 1K budget and batch size 320, throughput (1380.8 vs 1345.9) and memory (72.0 vs 72.5 GB) are essentially tied, but accuracy is 29.9 versus 42.2. BeaconKV is not buying accuracy with extra resources; it is keeping the right things inside the same budget.

One ablation is counterintuitive. On Qwen3-4B / AIME24 with a 2048 budget, more beacons is not better: the (4 beacon, 28 recent) configuration reaches the highest accuracy at 66.5, while the all-recent RPC configuration scores 51.3. A handful of beacons produces a double-digit gap, which suggests what is being captured is structure rather than volume.

So what

The contribution here is less "another speedup" than a methodological correction: a reasoning model generates its own context as it goes, which makes it a fundamentally different problem from conventional long-context processing. In the latter, the context is given and importance can be estimated statically. In the former, the model writes the context and then reads back what it wrote. Using a recent window as a proxy for future attention is unsound by construction in that setting.

Inference cost is becoming the dominant line item in deploying large models, and KV cache compression has drawn a dense stream of papers lately. BeaconKV's reminder is that the ceiling on compression depends less on how clever your scoring function is than on whether your observation window matches the model's actual attention dynamics. Code is open source (arXiv:2609.04971; implementation at aiha-lab/BeaconKV).

Models look back at the reasoning they have written. That fact may be worth more attention than any speedup multiple.