The real bottleneck for long context is not whether a model can read a long input, but whether the KV cache fits in memory. The dominant recipe so far: train with exact attention, then bolt a sparse KV eviction policy on top at inference time — so the memory mechanics differ between training and deployment. A paper submitted to arXiv on August 20, "Learning how to Forget" (2608.19920), flips that order: instead of patching after the fact, let the model learn while forgetting during fine-tuning.

Core idea: let weights co-adapt with the eviction policy

The paper is authored by Matthias Seeger and four colleagues, with the companion code open-sourced under AWS's awslabs organization. The key move: keep the KV cache eviction policy active throughout fine-tuning, so the weights adapt to the reality that information gets evicted — co-adapting with the policy. The abstract reports this often outperforms models trained with exact attention (sequence parallelism).

The reproduction bar is low: the README's example fine-tunes Qwen/Qwen3-4B-Instruct-2507 with LoRA on 128k-token sequences from the Helmet benchmark on a single A100 GPU with 40 GB of memory, using the h2o-torch-quantized8 cache policy with 16,384 slots; on an 8xA100 AWS p4d.24xlarge instance, the global batch reaches 32.

Engineering H2O into kernels

The leading eviction policy in the paper's experiments is H2O (arXiv:2306.14048): score KV entries by the cumulative sum of attention weights and evict the lowest scorer — which, per the README, is in a strong sense the LRU strategy from general caching.

The team also flags a gap the kernel ecosystem has overlooked: mainstream fast SDPA kernels do not return attention weights summed over the query axis, exactly what H2O-style scoring needs. The library's fix adds that return value to FlashInfer CUDA kernels and fills the computation with a Triton score-sum kernel; combined with 4/8-bit KV buffer quantization and CPU offloading, the README states 4-bit quantization cuts KV cache GPU memory requirements to a quarter.

Limitations are stated plainly

The README admits raw inference is not competitive with vLLM or SGLang, multi-device strategies are missing, and the library targets research and evaluation — though its support for advanced KV cache strategies like H2O is better than vLLM's. Another highlight is gradient computation: eviction decisions are logged into a replay log during the forward pass, then replayed through replay caches cell by cell during backward, paired with activation checkpointing, which is what makes long-sequence fine-tuning under sparse attention memory-feasible at all.

So what

The lasting idea here is not a benchmark number but a course correction: the next leg of long-context optimization may lie not in bigger windows but in aligning memory mechanics between training and deployment. Once eviction is part of training, forgetting stops being a defect and becomes a learnable skill. For teams shipping long-context workloads, a single-GPU reproducible setup plus fully open code means this route can be tested over a weekend (paper: https://arxiv.org/abs/2608.19920 , code: https://github.com/awslabs/keys_values ).