[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-rift-draft-kv-cache-compression":3,"topics-all":38,"news-related-02c96622-1409-4867-9f25-772936edc6cd":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"02c96622-1409-4867-9f25-772936edc6cd","滑动窗口反超SnapKV:NVIDIA重审投机解码KV压缩","NVIDIA 团队新论文命名投机解码的解码陈旧化:草稿模型的 KV cache 选择随输出变长而失效,同等预算下朴素滑动窗口反超 SnapKV。RIFT 用动静结合刷新策略加压缩感知训练,让 Qwen3.5-4B 草稿在约八分之一缓存预算下保留约 96% 接受长度,叠加训练后超过 98%。","投机解码(speculative decoding)已经成了推理侧的标准配置:小模型先猜一串 token,大模型一次验证多个,输出分布不变,把多步解码压进一次前向。但 reasoning 和 agent 应用把输入、输出同时拉长之后,这套体系里一个不起眼的部件开始顶到天花板——草稿模型的 KV cache。\n\n10 月 22 日提交到 arXiv 的论文《RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training》([arXiv:2610.31098](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31098))正面处理了这个问题。十位作者来自 NVIDIA,其中四位也是论文所用的 SPEED-Bench 基准的共同作者。\n\n## 先看一个反直觉结果\n\n论文的第一组实验就颠覆了一个默认认知。同样 8K 的 KV 预算,让 Qwen3.5-4B 给 Qwen3.5-122B 当草稿、跑 32K 输入的 SPEED-Bench:带重要性打分的 SnapKV,输给了\"无脑只留最近 token\"的滑动窗口(SWA)。随着生成推进,SnapKV 的平均接受长度(AL)下降 36.7%,滑动窗口反而提升 41%。\n\n机制层面的证据更直接:全缓存注意力中落在保留集合上的比例,SnapKV 从 67.0% 一路掉到 25.4%,滑动窗口从 43.5% 涨到 68.7%,两条曲线在中途交叉。作者把这种现象命名为 **decode staleness(解码陈旧化)**:prefill 阶段选出来的\"重要\"KV,到长输出阶段已经不再重要。\n\n## 为什么会陈旧\n\n论文给出两条归因。其一,投机解码里最近的解码前缀是目标模型验证过的输出,草稿模型盯住它才能保持与目标对齐;其二,reasoning 场景下最近 token 本身就是一路滚动的\"思维链摘要\"。消融实验很说明问题:关闭 reasoning 后,staleness gap 从 2.11 掉到 0.60 AL——推理链越长,陈旧化越狠。\n\n还有一个架构维度的发现:在 Nemotron3(hybrid-SSM)和 GPT-OSS(硬编码 SWA 交错全注意力)这类混合注意力模型上,SWA 压过 SnapKV;只有在全注意力的 Llama3.1 上 SnapKV 才保持优势。而 KV 压缩文献的评测恰恰大多建立在全注意力加短输出的设定上。\n\n## RIFT 的解法:三档策略加一次训练\n\n- **RIFT-stat**:最简单的静态版。把预算 B 按比例 ρ 切成\"最近 token 窗口 + 重要性 prompt 位置\"两块。ρ=0.5 时 AL 达 5.614,比 SnapKV 的 4.073 高出 37.8%,也高于纯 SWA 的 5.230。\n- **RIFT-dyn**:解码过程中周期性重打分、刷新选择,质量更高但要付额外延迟。\n- **RIFT-ada**:自适应刷新,比 dyn 少 51.1% 的刷新次数,AL 反而还高 0.122。\n- **压缩感知训练**:用稀疏注意力掩码 + LoRA + 蒸馏(CE+KL)做一轮 post-training,让草稿模型天生适应压缩缓存。Qwen3.5-4B 最高拿到 +16.19%,Nemotron3.5-Lightning 最高 +16.42%。\n\n合成成绩:SPEED-Bench 32K 输入、4× 压缩比下,Qwen3.5-4B 保留 96% 的全缓存 AL,Nemotron3.5-Lightning 保留 97%(给 Nemotron3-Ultra 当草稿);SnapKV 在两组设定里只有 63-67%。结合压缩感知训练后,32K 输入、4K 输出、8K 预算下接受长度保留超过 98%。对比 StreamingLLM、H2O、TriAttention 三个解码时基线,RIFT-dyn 和 RIFT-ada 在 32K\u002F100K 两档上下文、两种温度下全部占优。\n\n## 别急着上线,看清两个边界\n\n论文的 Limitations 写得诚实。第一,在纯 GPU 部署里,外部草稿模型的端到端加速本来就不如 MTP、EAGLE、DFlash 这类 tail-augmented 方案——RIFT 的真实靶场是异构推理:GPU 负责打分,SRAM 类低内存加速器只跑草稿,论文引用了 NVIDIA LPX 低延迟加速器技术博客作为这一方向的早期信号。第二,解码中重打分的开销在 GPU 上约占草稿解码延迟的 6.0%,搬到 SRAM 设备上这个比例还要重新量。\n\n对行业的信号比单篇结果更重要:混合注意力正在成为旗舰模型的默认架构,而 KV 压缩的整套评测基准大多还是全注意力时代的遗产。方法选型之前,先问两个问题——你的输出长度分布是什么,你的模型是哪种注意力结构。这两个问题的答案,可能比压缩方法本身的榜单名次更值钱。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31098","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"1f2c8198-9459-4bcc-927a-06371180fba2","en","Sliding Window Beats SnapKV: NVIDIA Rethinks Draft KV Cache","NVIDIA's RIFT paper names decode staleness: sliding windows beat SnapKV at equal KV budget; training lifts retention past 98%.","Speculative decoding has become a standard fixture on the inference side: a small draft model proposes a run of tokens, the large target model verifies them in a single forward pass, and the output distribution stays intact. But as reasoning and agentic workloads stretch both inputs and outputs, an unglamorous component is hitting the ceiling — the draft model's KV cache.\n\nThe paper \"RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training\" ([arXiv:2610.31098](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31098)), submitted on October 22, tackles this head-on. The ten-author team is from NVIDIA, four of whom also co-authored the SPEED-Bench benchmark used throughout the evaluation.\n\n## A counterintuitive result first\n\nThe opening experiment overturns a default assumption. With an identical 8K KV budget, Qwen3.5-4B drafting for Qwen3.5-122B on 32K-input SPEED-Bench: SnapKV, which carries importance scoring, loses to a naive sliding-window attention (SWA) that just keeps the most recent tokens. As generation proceeds, SnapKV's average acceptance length (AL) drops 36.7% while the sliding window improves 41%.\n\nThe mechanistic evidence is blunt: the share of full-cache attention mass falling on the retained set slides from 67.0% to 25.4% for SnapKV, while rising from 43.5% to 68.7% for SWA — the curves cross mid-generation. The authors name the phenomenon **decode staleness**: KVs selected as \"important\" during prefill stop mattering once outputs grow long.\n\n## Why selections go stale\n\nTwo attributed causes. First, in speculative decoding the recent decode prefix is target-verified output; the draft must track it to stay aligned. Second, in reasoning settings recent tokens act as a rolling summary of the chain of thought. The ablation is telling: disabling reasoning shrinks the staleness gap from 2.11 to 0.60 AL — the longer the reasoning chain, the harsher the staleness.\n\nThere is an architecture dimension too: on hybrid-attention models like Nemotron3 (hybrid-SSM) and GPT-OSS (hard-coded SWA interleaved with full attention), SWA beats SnapKV; only on full-attention Llama3.1 does SnapKV keep its edge. Yet most KV-compression benchmarks are built on full-attention models with short outputs.\n\n## RIFT: three tiers of policy plus one training pass\n\n- **RIFT-stat**: the static version. Budget B is split by ratio ρ between a recent-token window and importance-selected prompt positions. At ρ=0.5, AL reaches 5.614 versus 4.073 for SnapKV (+37.8%) and 5.230 for pure SWA.\n- **RIFT-dyn**: periodically rescore and refresh the selection during decoding; higher quality, extra latency.\n- **RIFT-ada**: adaptive refresh — 51.1% fewer refreshes than dyn while gaining 0.122 AL.\n- **Compressibility training**: a post-training pass with sparse attention masks, LoRA, and CE+KL distillation so the draft natively fits a compressed cache. Up to +16.19% for Qwen3.5-4B and +16.42% for Nemotron3.5-Lightning.\n\nBottom line: at 32K input and 4x compression, Qwen3.5-4B retains 96% of full-cache AL and Nemotron3.5-Lightning retains 97% (drafting for Nemotron3-Ultra), while SnapKV retains only 63-67%. With compressibility training, over 98% of full-cache acceptance length is retained on 32K inputs, 4K outputs, and an 8K budget. Against decode-time baselines StreamingLLM, H2O, and TriAttention, RIFT-dyn and RIFT-ada win across both 32K and 100K contexts at two temperatures.\n\n## Read the boundaries before adopting\n\nThe limitations section is candid. First, on pure GPU deployments, external draft models already lose end-to-end to tail-augmented schemes like MTP, EAGLE, and DFlash — RIFT's real target is heterogeneous inference, where the GPU computes selection scores and an SRAM-class, memory-constrained accelerator runs the draft; the paper cites NVIDIA's LPX low-latency accelerator blog as an early signal. Second, decode-time rescoring costs up to 6.0% of draft decode latency on GPU, a number that must be re-measured on SRAM devices.\n\nThe signal for the industry outweighs any single result: hybrid attention is becoming the default for flagship models, while the KV-compression benchmark suite is largely an heirloom of the full-attention era. Before picking a method, ask two questions — what does your output-length distribution look like, and which attention architecture are you running? Those two answers may be worth more than any leaderboard rank.","rift-draft-kv-cache-compression","2026-10-23T15:20:00Z","2026-10-23T15:11:33.574896Z","2026-10-23T15:11:33.574905Z",true,"agent",2,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"a046b158-396f-4db9-a8d3-22d8c193a166","清华把KV缓存当神经网络训练:32倍压缩准确率96.7%","latent-compaction-test-time-kv","2026-10-15T17:08:40+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"dab2e487-cfba-4d75-a93f-eff133476348","剪枝校准对不上解码分布?西湖蚂蚁让稀疏解码提速48%","sparsedecoding-decoding-aware-pruning","2026-10-10T19:09:23+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"c83af54b-79ed-445c-9482-07d98c26c36b","BeaconKV:长推理会回头看,只压最近窗口的 KV 缓存注定丢东西","beaconkv-beacon-query-kv-cache-compression","2026-09-09T11:25:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"bce9fc16-d31a-49be-b17b-f144619a58e2","LatentPress:上下文压成软令牌直读，7.7 倍压缩反超原文，训练仅动 0.1% 参数","latentpress-soft-token-context-compression","2026-09-05T19:06:09+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"d31bc388-b6c7-41a5-a6e9-6f00657c7616","加GPU还是压KV缓存？arXiv论文：压缩省钱1.2到2倍，但36B是道坎","tensor-parallelism-vs-kv-compression-cost","2026-08-30T17:10:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"d2c56430-f9a7-4844-bfab-a8651279a70c","ResKV 不再把 KV 缓存压缩等同于删词：给被淘汰的信息留一份残差账本","reskv-residual-kv-cache-compression","2026-08-03T10:43:23+00:00"]