[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-random-attention-kv-cache-eviction":3,"news-related-7a6d28b6-65da-4a29-96b1-dedb9894de97":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"7a6d28b6-65da-4a29-96b1-dedb9894de97","随机驱逐追平最强打分器:Salesforce 重写 KV Cache 压缩常识","Salesforce AI Research 论文发现:推理模型的 KV Cache 压缩中,精心设计的打分选择信号几乎不起作用。只保 prompt、对推理 token 均匀随机驱逐,在 4 个模型 6 个任务上追平最强选择性方法,vLLM 吞吐高 32%-43%,原因是推理链自带两层冗余。","长推理模型的显存账,最后都算在 KV Cache 头上。模型每生成一个 token,注意力机制都要把之前的键值状态缓存下来;一条几万 token 的思维链跑下来,KV Cache 成了严重的内存瓶颈——这是 Salesforce AI Research 新论文(arXiv:2609.03430,9 月 3 日提交)开篇给出的背景。而这个方向已有整整一族方法,思路高度一致:给缓存里每个 token 打个分,估计它\"以后还有没有用\",留下分高的,扔掉分低的。论文的核心结论是:打分这个动作本身,几乎没有贡献。\n\n## 随机驱逐,追平最强打分器\n\n论文提出的方法叫 Random Attention,规则简单到不像一个方法:prompt 部分完整保留,生成的推理 token 在每个注意力头内部均匀随机驱逐,预算用完为止,外加一小段最近窗口兜底——不读注意力分数,不算 value 统计,不需要任何校准数据。作者在 Qwen3-4B\u002F14B\u002F32B 和 Phi-4-reasoning 四个模型、MATH-500、GPQA-Diamond、AIME、HMMT、LiveCodeBench 等六个推理任务上做了对照:同等缓存预算下,随机驱逐追平了 SnapKV、R-KV、VaSE、TriAttention 这些精心设计的选择性方法;更关键的是部署端数字——在 vLLM 服务栈里,它的吞吐比所对比的最强驱逐器高 32%-43%。省掉打分计算,直接省出了吞吐。\n\n## 为什么随机就够:推理链自带两层冗余\n\n论文最有价值的部分不是\"随机也行\"这个结论,而是机制解释。作者做了控制实验,拆出两层原因。其一,缓存里真正脆弱的是 prompt:各打分器之间的性能差距,大半可以归结为\"它的选择信号碰巧有没有保住 prompt\"——prompt 安全了,后面的推理 token 怎么选差别不大。其二,推理链自身有双保险:文本层面,模型边推理边重述自己还需要的信息;注意力头层面,每个头都持有一份推理链的副本。两层冗余叠加,随机抽取也总能留下足够份数的关键信息。\n\n换句话说:不是随机有多聪明,而是此前的打分器一直在为\"碰巧保住 prompt\"和\"冗余兜底\"这两件本来不需要打分的事邀功。选择信号贡献近乎为零,才是这批方法真正共享的隐性前提。\n\n## 工程视角:卖点在成本结构\n\n对做推理服务的人,这篇论文的价值不在精度,在成本。项目 README 写得很直白:一轮驱逐的开销只剩下压缩动作本身;对比之下,TriAttention 需要逐模型的校准统计,部署门槛在模型之外。代码已按 Apache 2.0 开源,仓库包含驱逐引擎、评测 harness、显著性检验、vLLM 移植与机制研究工具链,论文实验在 8 张 H200(141GB)上完成。在 Hugging Face 论文页,它两天内已收获逾百个 upvote。\n\n照例泼冷水:这是单团队结果,发布不到两天,还需要独立复现来检验;研究对象是长思维链的推理场景,结论不能直接外推到所有 KV Cache 压缩场景;32%-43% 的吞吐提升对照的是\"最强选择性驱逐器\",与不压缩的满血缓存相比是另一本账。\n\n但对整个方向的提醒是实打实的:下次再看到某篇论文宣称新的 KV Cache 打分算法涨了多少点,第一个该问的问题是——随机基线,跑过了吗?\n\n参考:arXiv:2609.03430(https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.03430);代码:github.com\u002FSalesforceAIResearch\u002FRandom-Attention","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.03430","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"d13ea8a7-eb02-49d8-bac0-1ae46b62d48b","en","Random eviction matches top scorers: Salesforce rewrites KV cache","Random KV cache eviction—keep the prompt, no scoring—matches the strongest selector on 4 models and 6 tasks, with 32-43% higher vLLM throughput.","The memory bill for long-reasoning models ultimately lands on the KV cache. Every generated token forces the attention mechanism to cache its key-value states; run a tens-of-thousands-token chain of thought, and the KV cache becomes a severe memory bottleneck — that is the opening framing of a new paper (arXiv:2609.03430, submitted September 3) from Salesforce AI Research. The field already has a whole family of methods that share one paradigm: score each cached token by some estimate of how much it will matter later, keep the top-scoring ones, evict the rest. The paper's core finding: the selection signal itself contributes almost nothing.\n\n## Random eviction matches the strongest scorer\n\nThe proposed method, Random Attention, is almost embarrassingly simple: keep the prompt intact, then evict generated reasoning tokens uniformly at random within each attention head up to the budget, plus a short recency window — reading no attention scores, no value statistics, and no calibration data. Across four models (Qwen3-4B\u002F14B\u002F32B and Phi-4-reasoning) and six reasoning tasks (MATH-500, GPQA-Diamond, AIME, HMMT, LiveCodeBench), at matched budgets, random eviction matches carefully designed selective methods like SnapKV, R-KV, VaSE, and TriAttention. The deployment-side number matters more: in a vLLM serving stack, it delivers 32-43% higher throughput than the strongest prior evictor. Skipping the scoring compute converts directly into throughput.\n\n## Why randomness suffices: reasoning traces carry two layers of redundancy\n\nThe most valuable part of the paper is not the \"random works too\" headline but the mechanism behind it. Controlled experiments isolate two causes. First, the fragile part of the cache is the prompt: most of the performance gap between selectors comes down to whether their selection signal happened to keep the prompt — once the prompt is safe, how you pick among reasoning tokens matters little. Second, the reasoning trace protects itself with double insurance: at the text level, the model restates what it still needs as it works; at the attention-head level, each head keeps its own copy of the trace. With both layers stacked, a random draw reliably retains enough copies of whatever the model still needs.\n\nPut differently: it is not that randomness is clever — it is that previous scorers have been taking credit for two things that never needed scoring, namely \"happening to keep the prompt\" and \"redundancy doing the backup.\" A near-zero contribution from the selection signal is the hidden premise this whole family of methods shares.\n\n## The engineering angle: the selling point is the cost structure\n\nFor inference-serving engineers, the value here is not accuracy but cost. The project README states it plainly: an eviction round costs only the compaction itself, whereas TriAttention requires per-model calibration statistics — a deployment burden beyond the model. The code is open-sourced under Apache 2.0, shipping the eviction engine, evaluation harness, significance tests, the vLLM port, and the mechanism-study tooling; the paper's experiments ran on 8x H200 (141GB). On the Hugging Face papers page it has drawn over a hundred upvotes within two days.\n\nThe usual cold water: this is a single-team result, out for less than two days, and awaits independent replication; the study targets long-chain-of-thought reasoning scenarios, so the conclusion should not be blindly extrapolated to every KV cache compression setting; and the 32-43% throughput gain is measured against the strongest selective evictor — versus a fully uncompressed cache it is a different accounting.\n\nStill, the wake-up call for the field is real: the next time a paper claims its new KV cache scoring algorithm gains a few points, the first question to ask is — did you run the random baseline?\n\nReference: arXiv:2609.03430 (https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.03430); code: github.com\u002FSalesforceAIResearch\u002FRandom-Attention","random-attention-kv-cache-eviction","2026-09-04T19:08:26Z","2026-09-04T19:08:35.511680Z","2026-09-04T19:08:35.511700Z",true,"agent",34,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"199cd4ef-f092-45a5-8635-91778dd2bce2","编译即训练：一句规约炼出 83.6% 准确率的本地神经函数，教师模型只用一次","compile-by-training-neural-functions","2026-09-04T23:08:03+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"44a035c8-b8a3-48e5-af4f-c76323dac7b5","RWKV7-G1j 13.3B 开源:不用注意力,每 token 推理成本是常数","rwkv7-g1j-13b-attention-free","2026-09-03T13:14:19+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"ff3b7c9f-e338-4af9-a562-4fda8b1bc929","vLLM 0.28 发布:584 项提交押注 Kimi K3,每卡省 17GB 显存","vllm-0-28-kimi-k3-optimization","2026-08-29T19:05:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"92433e6b-113a-4ada-af77-fbb8995a9850","LFM2.5-DSpark 开源:300M 草稿模型让端侧推理快 2.87 倍,输出零损耗","lfm2-5-dspark-draft-models","2026-08-21T21:10:00+00:00"]