JoLT compresses KV cache to one-third with Tucker residuals

The #1 bottleneck of Transformer long-context inference is the KV cache — it expands with batch, context length, and depth, eating more memory than the model weights themselves. Mainstream compression routes have two paths: low-rank decomposition only looks at the 2D slice of the cache, and quantization methods only compress the bit width — neither uses the "heads × tokens × features" third-order tensor structure thoroughly. arXiv:2607.12550 (2026-07-14, Krishnan & Schulz) proposes JoLT, taking a third path: treating each layer's KV cache as a third-order tensor, performing partial Tucker decomposition only on the token and feature axes (keeping the heads and layer axes), and then using a Johnson-Lindenstrauss-rotated low-bit residual to make up for the truncated energy. A Lagrangian dual unifies the allocation of Tucker rank and residual bit width, with per-layer group and K/V separated budgets. The measured results are clean. On Mistral-7B-v0.3 (GQA) and LLaMA-2-13B (MHA), after 2–3× compression, perplexity, GSM8K, and RULER retrieval are all within the statistical noise of the uncompressed baseline; at 2× the relative Frobenius error is only 0.009 (K) / 0.006 (V), an order of magnitude lower than cross-layer SVD and 4-bit quantization. The companion FlashJoLT randomized SVD variant cuts compression time by another 5–13×. Two details worth highlighting: first, partial Tucker deliberately avoids low-rank projection on the heads axis — in multi-head attention heads are inherently "each on its own patch", flattening them directly destroys expressiveness; second, the JL rotation whitens the low-bit residual, making quantization error close to an independent uniform distribution, which is the key to holding accuracy at the low-bit tier. On the deployment side, long-context Agents and batch inference services are the direct beneficiaries: the number of concurrent sessions on the same 80GB H100 can be directly doubled, without retraining. The code isn't public yet; the engineering details will need to be revisited after open-sourcing.