RAG pipelines, long-term conversational memory, and tool-using agents assemble each prompt from retrieved chunks. Because the combination changes per request, prefix caching rarely applies, so many systems use non-prefix KV-cache reuse: encode each chunk once, store its keys and values, and concatenate the stored caches at query time, skipping the prefill over the retrieved context. A new arXiv paper (2611.06397) works out what that saving costs in accuracy: up to all of the context-dependent questions a model would otherwise answer.

How much reuse loses

The authors evaluate Qwen3-8B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.2 on four datasets — LoCoMo (dialogue memory), NQ-Open and TriviaQA (open-domain QA), and Gorilla (API documentation) — twelve settings in total, scoring only questions that full prefill answers and that require the retrieved context. Position-corrected reuse loses 19-100% of those questions.

The loss depends on the distractors. It persists with off-domain prose around the target; unrelated same-corpus chunks make it significantly worse in nine of twelve settings; passages similar to the target reduce accuracy further in all six QA settings, significantly in five. A model fine-tuned by Ma et al. (2025) to encode documents as independent blocks still loses 60% of these questions on NQ-Open, mostly from the similar chunks. In the paper's example, Qwen3-8B answers correctly from the full-prefill cache but names the actress from a similar passage once the same passages are reused.

Where the repair budget belongs

Existing repairs split into token-level selectors (CacheBlend, EPIC, ProphetKV, CacheTune) that recompute scattered tokens online, and surrogate-prefix methods like SurroKV. The paper asks a simpler question: with a small budget, whom should you repair?

The answer is the target chunk. Recoverable damage is spread over the distractors, so restoring a few of them recovers little, while the target chunk is a median of only 2.5-3.3% of the context — inside a 5% budget. The mechanism is contextual compatibility: recomputing the target against the reused cache, so that it is encoded in the presence of the other chunks, beats inserting its full-prefill KV in all seven settings tested (four significant after Holm's correction), even though the recomputed KV stays at a median relative distance of 0.23-0.30 from the full-prefill KV. What matters is not approaching the reference values but encoding consistency.

When you do not know the target

Question attention over reused caches selects the target in at most 18.8% of development questions (12.5% for the other detector), while summed attention locks onto the chunk just before the question in 58-100% of them; the retriever ranks the target first on only 47-64% of questions. The deployment recipe: move the retriever's top chunk to the position just before the question and recompute it — at the same 5% budget this significantly beats InfoFlow-style reordering in six of nine QA and API-documentation settings, and with the target known, 5%-budget recomputation beats token-level selectors in all nine.

The boundary: dialogue memory

On LoCoMo the conclusion does not transfer: a whole session fits within the 5% budget on only 11-21% of questions (89-94% at a 10% budget), and recomputing the target session changes retention by -0.04 to +0.03 versus a token-level selector — never significant. When chunks are too big for the budget, the target-first accounting breaks down.

So what

The paper turns cache repair from a craft into a budget-allocation problem: measure your retriever's top-1 hit rate before choosing a repair strategy, and prefer recomputing against the reused cache over copying full-prefill KV. If you run retrieval inference, ask one question: has the prefill compute you saved ever been audited against the questions you lost? (Paper: https://arxiv.org/abs/2611.06397)