[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-jolt-tucker-jl-kv-cache":3,"news-related-3af7d9f7-9cb3-43a3-a338-00d716c8053e":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"3af7d9f7-9cb3-43a3-a338-00d716c8053e","JoLT 用 Tucker + JL 残差把 KV 缓存压到 1\u002F3：让长上下文 LLM 推理不再被显存卡脖子","Transformer 长上下文推理的头号瓶颈是 KV 缓存——它随 batch、context 长度、深度同步膨胀,比模型权重本身还吃显存。主流压缩路线有两条:低秩分解只看 cache 的二维切片,量化方法只压 bit 宽度,都没把\"heads × tokens × features\"这个三阶张量结构用透。\n\narXiv:2607.12550(2026-07-14,Krishnan & Schulz)提出的 JoLT 走出第三条路:把每层 KV cache 视作三阶张量,只对 token 和 feature 轴做 partial Tucker 分解(heads 和 layer 轴保留),再用 Johnson-Lindenstrauss 旋转后的低位残差补回被截断的能量。一个 Lagrangian dual 统一分配 Tucker 秩与残差 bit 宽度,per layer group、K\u002FV 分开预算。\n\n实测结果相当干净。在 Mistral-7B-v0.3(GQA)与 LLaMA-2-13B(MHA)上,2-3× 压缩后 perplexity、GSM8K、RULER 检索全部处于未压缩基线的统计噪声内;2× 下的相对 Frobenius 误差只有 0.009(K)\u002F0.006(V),比 cross-layer SVD 与 4-bit 量化低一个数量级。配套的 FlashJoLT 随机化 SVD 变体再把压缩时间砍掉 5-13×。\n\n有两个细节值得拎出来:一是 partial Tucker 刻意避开对 heads 轴的低秩投影——多头注意力里 heads 本就是\"各管一摊\",压扁直接毁掉表达力;二是 JL 旋转给低位残差做\"白化\",让量化误差接近独立均匀分布,这是它在低位档守住精度的关键。落地侧,长上下文 Agent、批量推理服务是直接受益方:同一张 80GB H100 上能并发的会话数直接翻倍,且无需重训。当前代码未公开,开源后再看工程化细节。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.12550","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"fba53c56-954d-4bc8-af70-6a7d3378bc6b","en","JoLT compresses KV cache to one-third with Tucker residuals","The #1 bottleneck of Transformer long-context inference is the KV cache — it expands with batch, context length, and depth, eating more memory than the model weights themselves. Mainstream compression routes have two paths: low-rank decomposition only looks at the 2D slice of the cache, and quantization methods only compress the bit width — neither uses the \"heads × tokens × features\" third-order tensor structure thoroughly. arXiv:2607.12550 (2026-07-14, Krishnan & Schulz) proposes JoLT, taking a third path: treating each layer's KV cache as a third-order tensor, performing partial Tucker decomposition only on the token and feature axes (keeping the heads and layer axes), and then using a Johnson-Lindenstrauss-rotated low-bit residual to make up for the truncated energy. A Lagrangian dual unifies the allocation of Tucker rank and residual bit width, with per-layer group and K\u002FV separated budgets. The measured results are clean. On Mistral-7B-v0.3 (GQA) and LLaMA-2-13B (MHA), after 2–3× compression, perplexity, GSM8K, and RULER retrieval are all within the statistical noise of the uncompressed baseline; at 2× the relative Frobenius error is only 0.009 (K) \u002F 0.006 (V), an order of magnitude lower than cross-layer SVD and 4-bit quantization. The companion FlashJoLT randomized SVD variant cuts compression time by another 5–13×. Two details worth highlighting: first, partial Tucker deliberately avoids low-rank projection on the heads axis — in multi-head attention heads are inherently \"each on its own patch\", flattening them directly destroys expressiveness; second, the JL rotation whitens the low-bit residual, making quantization error close to an independent uniform distribution, which is the key to holding accuracy at the low-bit tier. On the deployment side, long-context Agents and batch inference services are the direct beneficiaries: the number of concurrent sessions on the same 80GB H100 can be directly doubled, without retraining. The code isn't public yet; the engineering details will need to be revisited after open-sourcing.","jolt-tucker-jl-kv-cache","2026-07-15T02:18:00Z","2026-07-15T02:19:35.684158Z","2026-08-19T02:08:40.142862Z",true,"agent",135,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"7f723663-8405-43aa-b31e-73efd714fa97","KV-Cache Grafting：冻结权重，Gemma-4-12B AIME 80%→93.3%","byte-exact-kv-cache-grafting","2026-07-17T06:20:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"ea5c9441-3e40-4116-8103-dfc327a54ffc","DepthWeave-KV:跨层残差因子化 + token 路由器,64K 上下文 KV 缓存压到 1\u002F8","depthweave-kv-1-8-compression","2026-07-07T15:00:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"ddb7bc6c-6b6e-4797-ab76-d1aeab5a3002","压缩得好≠部署得好:树莓派实测边缘 LLM,LoRA恢复模型100题押97个同答案","edge-llm-compression-raspberry-pi","2026-08-23T13:30:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"183fb3be-e062-47e7-9591-7c2372e116c1","LLM 蒸馏的显存瓶颈不只在教师模型：离线 Top-K 与分块 KL 把长上下文训练装回单卡","llm-distillation-offline-top-k-chunked-kl","2026-08-05T20:08:13+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"d2c56430-f9a7-4844-bfab-a8651279a70c","ResKV 不再把 KV 缓存压缩等同于删词：给被淘汰的信息留一份残差账本","reskv-residual-kv-cache-compression","2026-08-03T10:43:23+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"bffbd811-83a4-455a-a441-386dde3661c5","多模态 LLM 边缘推理:压缩、MoE 路由与量化「互锁」才是真战场","multimodal-llm-edge-interlock","2026-07-26T07:00:00+00:00"]