Speculative decoding has become a standard fixture on the inference side: a small draft model proposes a run of tokens, the large target model verifies them in a single forward pass, and the output distribution stays intact. But as reasoning and agentic workloads stretch both inputs and outputs, an unglamorous component is hitting the ceiling — the draft model's KV cache.

The paper "RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training" (arXiv:2610.31098), submitted on October 22, tackles this head-on. The ten-author team is from NVIDIA, four of whom also co-authored the SPEED-Bench benchmark used throughout the evaluation.

A counterintuitive result first

The opening experiment overturns a default assumption. With an identical 8K KV budget, Qwen3.5-4B drafting for Qwen3.5-122B on 32K-input SPEED-Bench: SnapKV, which carries importance scoring, loses to a naive sliding-window attention (SWA) that just keeps the most recent tokens. As generation proceeds, SnapKV's average acceptance length (AL) drops 36.7% while the sliding window improves 41%.

The mechanistic evidence is blunt: the share of full-cache attention mass falling on the retained set slides from 67.0% to 25.4% for SnapKV, while rising from 43.5% to 68.7% for SWA — the curves cross mid-generation. The authors name the phenomenon decode staleness: KVs selected as "important" during prefill stop mattering once outputs grow long.

Why selections go stale

Two attributed causes. First, in speculative decoding the recent decode prefix is target-verified output; the draft must track it to stay aligned. Second, in reasoning settings recent tokens act as a rolling summary of the chain of thought. The ablation is telling: disabling reasoning shrinks the staleness gap from 2.11 to 0.60 AL — the longer the reasoning chain, the harsher the staleness.

There is an architecture dimension too: on hybrid-attention models like Nemotron3 (hybrid-SSM) and GPT-OSS (hard-coded SWA interleaved with full attention), SWA beats SnapKV; only on full-attention Llama3.1 does SnapKV keep its edge. Yet most KV-compression benchmarks are built on full-attention models with short outputs.

RIFT: three tiers of policy plus one training pass

  • RIFT-stat: the static version. Budget B is split by ratio ρ between a recent-token window and importance-selected prompt positions. At ρ=0.5, AL reaches 5.614 versus 4.073 for SnapKV (+37.8%) and 5.230 for pure SWA.
  • RIFT-dyn: periodically rescore and refresh the selection during decoding; higher quality, extra latency.
  • RIFT-ada: adaptive refresh — 51.1% fewer refreshes than dyn while gaining 0.122 AL.
  • Compressibility training: a post-training pass with sparse attention masks, LoRA, and CE+KL distillation so the draft natively fits a compressed cache. Up to +16.19% for Qwen3.5-4B and +16.42% for Nemotron3.5-Lightning.

Bottom line: at 32K input and 4x compression, Qwen3.5-4B retains 96% of full-cache AL and Nemotron3.5-Lightning retains 97% (drafting for Nemotron3-Ultra), while SnapKV retains only 63-67%. With compressibility training, over 98% of full-cache acceptance length is retained on 32K inputs, 4K outputs, and an 8K budget. Against decode-time baselines StreamingLLM, H2O, and TriAttention, RIFT-dyn and RIFT-ada win across both 32K and 100K contexts at two temperatures.

Read the boundaries before adopting

The limitations section is candid. First, on pure GPU deployments, external draft models already lose end-to-end to tail-augmented schemes like MTP, EAGLE, and DFlash — RIFT's real target is heterogeneous inference, where the GPU computes selection scores and an SRAM-class, memory-constrained accelerator runs the draft; the paper cites NVIDIA's LPX low-latency accelerator blog as an early signal. Second, decode-time rescoring costs up to 6.0% of draft decode latency on GPU, a number that must be re-measured on SRAM devices.

The signal for the industry outweighs any single result: hybrid attention is becoming the default for flagship models, while the KV-compression benchmark suite is largely an heirloom of the full-attention era. Before picking a method, ask two questions — what does your output-length distribution look like, and which attention architecture are you running? Those two answers may be worth more than any leaderboard rank.