Feed a long document to a large model and the first thing to burst is not the parameters but the KV cache: every extra token adds another row of key-value entries, inflating memory and attention cost. The mainstream answers of recent years have been compression strategies — evict unimportant entries, or quantize the cache to lower precision. A paper submitted to arXiv on October 14 by Tsinghua's THUNLP team, Latent Compaction, takes a different route: stop picking and discarding, and instead treat the KV cache as a small trainable neural network, using test-time training to learn a compact cache for each document.
Treat the Cache as a Neural Network, Train in Two Stages
The method runs in two steps. Initialization selects seed entries using three signals: reconstruction importance, teacher attention, and key-space diversity. Stage one fits the compact cache's attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. Stage two jointly optimizes the entire cache through end-to-end distillation from the full-cache model. The default schedule in the GitHub README is 1,000 updates per stage.
Two constraints hold throughout: model weights stay frozen — only the cache moves — and training uses only generic prompts about the document, with no access to downstream questions or answers. The compacted cache is therefore a document-specific asset: train once, then reuse it to answer many unseen questions.
96.7% versus 40.0% at 32x Compression
The paper evaluates on RULER, LongBench, and Phonebook Lookup against budget-matched compression baselines. Its headline number is on RULER: 96.7% retrieval accuracy at 32x compression, while budget-matched KVZip+ reaches 40.0%. The reproduction package is thorough — KVZip+, Attention Matching, SnapKV, PyramidKV, and a text-summary baseline all run on a single runner, with reproduction commands in the README. Phonebook experiments span context lengths from 4,096 to 32,768 and compression ratios from 2x to 256x. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5; the quick-start example uses Qwen3-1.7B.
Cost and Where It Sits in the Race
There is no free lunch. Each document pays a one-time test-time training cost (1,000 updates per stage by default) in exchange for every future question sharing a cache 1/32 the size. For single-shot question answering it loses money; for workloads that query the same long document repeatedly — support knowledge bases, contract repositories, codebases — the amortization logic holds immediately.
Placed on the race map: the quantization camp bets on representational precision, eviction methods (the SnapKV line) bet on selection heuristics, and Latent Compaction bets that the cache itself has learnable structure. While most test-time-training work aims at boosting reasoning, the Tsinghua team points the technique at compression, giving the line a concrete engineering outlet. The code is MIT-licensed, and the 19-page paper is at arXiv:2610.20582 (https://arxiv.org/abs/2610.20582), with the repository at github.com/THUNLP/latent-compaction.
The one-line takeaway: once the cache stops being an accessory and becomes an optimizable object, one term in the long-context cost formula is rewritten — the only question left is whether each document is worth 2,000 training steps.