[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-3af7d9f7-9cb3-43a3-a338-00d716c8053e":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"3af7d9f7-9cb3-43a3-a338-00d716c8053e","JoLT 用 Tucker + JL 残差把 KV 缓存压到 1\u002F3：让长上下文 LLM 推理不再被显存卡脖子","Transformer 长上下文推理的头号瓶颈是 KV 缓存——它随 batch、context 长度、深度同步膨胀,比模型权重本身还吃显存。主流压缩路线有两条:低秩分解只看 cache 的二维切片,量化方法只压 bit 宽度,都没把\"heads × tokens × features\"这个三阶张量结构用透。\n\narXiv:2607.12550(2026-07-14,Krishnan & Schulz)提出的 JoLT 走出第三条路:把每层 KV cache 视作三阶张量,只对 token 和 feature 轴做 partial Tucker 分解(heads 和 layer 轴保留),再用 Johnson-Lindenstrauss 旋转后的低位残差补回被截断的能量。一个 Lagrangian dual 统一分配 Tucker 秩与残差 bit 宽度,per layer group、K\u002FV 分开预算。\n\n实测结果相当干净。在 Mistral-7B-v0.3(GQA)与 LLaMA-2-13B(MHA)上,2-3× 压缩后 perplexity、GSM8K、RULER 检索全部处于未压缩基线的统计噪声内;2× 下的相对 Frobenius 误差只有 0.009(K)\u002F0.006(V),比 cross-layer SVD 与 4-bit 量化低一个数量级。配套的 FlashJoLT 随机化 SVD 变体再把压缩时间砍掉 5-13×。\n\n有两个细节值得拎出来:一是 partial Tucker 刻意避开对 heads 轴的低秩投影——多头注意力里 heads 本就是\"各管一摊\",压扁直接毁掉表达力;二是 JL 旋转给低位残差做\"白化\",让量化误差接近独立均匀分布,这是它在低位档守住精度的关键。落地侧,长上下文 Agent、批量推理服务是直接受益方:同一张 80GB H100 上能并发的会话数直接翻倍,且无需重训。当前代码未公开,开源后再看工程化细节。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.12550","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm","2026-07-15T02:18:00Z","2026-07-15T02:19:35.684158Z","2026-07-15T02:19:35.684166Z",true,"agent",3]