An agent working through a long-horizon task keeps calling tools and accumulating observations. As the trajectory grows, KV cache I/O and storage quickly become the dominant cost. The trajectory can even exceed the model's maximum context length, meaning an agent must actively control how its cache grows. The complication is that the mainstream remedy, "context engineering," shrinks that history by trimming or summarizing it. But this creates a structural dilemma: modifying the trajectory prefix invalidates every downstream cached state, so retained context has to be re-prefilled. And trimming context hard enough to avoid re-prefill costs degrades task accuracy.

The paper puts that dilemma front and center, then runs a controlled comparison across three models and four agent benchmarks, compressing the result into one set of numbers: average benchmark scores improve by 1.7 points, average cost drops by 27.7%, output throughput reaches 1.2x, and P95 request latency sees a 1.5x speedup.

Changing the cache instead of the text

The paper's move is to shift the control plane. Rather than rewriting the text trajectory, the agent operates directly on its own active KV cache, formalizing two operators.

  • drop excludes selected cache entries from future attention, so those keys and values stop participating in computation while remaining in the system.
  • repos adjusts the position encodings of retained keys, compacting their position indices so the position sequence stays in logical order, and resets the position index for the next append.

Working with rotary position embeddings (RoPE), repos moves a cached key from position p to position q via the rotation update K ← R(q)·R(p)⁻¹·K, while cached values remain unchanged.

The payoff is direct: everything except the dropped entries can be reused without re-prefilling. That breaks the binding between "saving cost" and "hurting accuracy" that characterizes context engineering — and because the text is never modified, no prefix is invalidated.

The paper also quantifies the problem with traditional approaches. On the first 100 BCP tasks, with four trials per task and at most 100 agent rounds, trim-based context engineering retaining only the latest 12 rounds of tool responses costs 1.85x the full-context baseline. Summary-based context engineering at a 48k threshold reduces cost by about 7% but lowers accuracy by 5 percentage points. Dropping older tool-response cache directly, by contrast, reuses the remaining cache while improving both accuracy and cost.

The real engineering obstacle: history dependence

The paper flags a subtlety that is easy to miss but fatal for serving systems: history dependence. The same token sequence can yield different cache states under different cache operation sequences. That means conventional prefix matching is insufficient for correct cache reuse.

AKV's system side co-designs three things in response: a stateless interface, an extended radix tree, and segment-batch attention. Externally it exposes a stateless interface compatible with existing serving APIs, while internally it manages stateful cache reuse, eviction, and restoration.

On the agent side, a CachePolicy decides what to drop and when to compact positions, with three built-in trigger styles: turn-based, token-based, and model-based, where the model decides what to drop and whether to reposition through a CacheTool. The system side executes those cache operations through SGLang's OpenAI-compatible Chat Completions API.

The headline configuration the paper reports is a tool-response rolling-drop policy retaining the KV states of the latest 12 tool responses, with its repos threshold matched to each baseline's summarization threshold.

How the experiments were run

At the model level, evaluation covers three LLMs: MiniMax-M2.7, GPT-OSS-120B, and MiroThinker-1.7-Mini. The benchmarks span four agent tasks: BrowseComp-Plus (BCP), xbench-DeepSearch-2510 (xbench-DS), the text-only 103-task GAIA validation subset (GAIA-Text) for deep research, and SWE-Verified for software engineering.

On the serving side, baselines run on SGLang, which is also AKV's base serving system, using eight NVIDIA A800-80GB GPUs — four for prefill and four for decode, with tensor parallelism of 4. Because the A800 lacks native MXFP4 support, all LLM weights are set to BF16.

In the controlled comparison, AKV achieves the highest average success rate across all three models, improving the average success rate by 1.3–2.1% and 0.9–8.7% against two baselines respectively. Inference cost falls by 27.7% relative to Standard and 21.9% relative to Text Engineering, averaged across all three models and four benchmarks. Letting the policy iterate recursively on held-out tasks reduces cost by a further 7.1% compared with expert-designed policies.

My read

The value of this paper is not any single benchmark number but the way it frames "who owns the cache." KV cache was previously treated as an internal implementation detail of the serving system — the system decides what to reuse and what to evict. This paper argues that the agent and its harness actually know better than the system which intermediate steps have become useless, and therefore should be able to issue cache operations directly, with the system executing them efficiently.

The history-dependence finding deserves particular attention. If different cache operation sequences can produce different cache states, then many incremental-caching mechanisms built on prefix matching are unreliable in exactly the high-concurrency, heavily branched, long-trajectory settings agents create. AKV treats it as a first-class citizen, which also explains why it needs a purpose-built engine rather than a configuration flag on an existing system.

If that framing holds, the interface contract between agent harnesses and serving engines may become an engineering standard worth watching. Both paper and code are public: the repository is at https://github.com/thu-nics/AKV and the paper at https://arxiv.org/abs/2610.28577 .