[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sparpr-kv-cache-recovery-consumer-gpu":3,"topics-all":38,"news-related-be76fca7-bc02-4a76-a58f-17a661b82b60":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"be76fca7-bc02-4a76-a58f-17a661b82b60","SparPR:消费级显卡的KV缓存回收术","arXiv 论文 SparPR 把消费级显卡跑大模型的 KV 缓存瓶颈拆成三个在线决策:哪些块值得留、留多少、重算还是重载。不改权重也不用离线训练,单台 RTX 3070 Ti 8GB 笔记本上 1k\u002F2k\u002F4k token 延迟分别降低 31%、52%、63%,并在 FlexGen 超时的 8k 场景跑完。","在消费级显卡上跑大模型，真正的天花板往往不是权重，而是 KV 缓存。\n\n权重是一次性加载的，KV 缓存却随着每解码一个 token 就往上涨。数据中心 GPU 靠大容量和高带宽能吸收这份开销，换到一台 8GB 显存的笔记本上，同样的开销会直接翻译成吞吐量下滑、延迟飙升，以及远在模型标称上下文窗口之前就先撞上的 OOM。\n\narXiv 在 2026 年 10 月 13 日放出的一篇论文（SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware，https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.18140）尝试处理这个老问题。它不做压缩也不做量化，而是把「缓存装不下」拆成三个可以在线做出的运行时决策。\n\n## 三个决策，而不是一个压缩率\n\n论文把已有工作归成三条互不相通的线：FlexGen、LeoAM 这类分层卸载系统回答「能不能跑起来」；StreamingLLM、H2O 这类缓存保留方法回答「该留哪些信息」；Quest、ProphetKV 这类局部恢复方法回答「怎么把它弄回来」。SparPR 的主张是，这三个问题本该串在同一个运行时循环里。\n\n**Query-Aware 的 key-block 选择。** 给每个历史块预计算一个紧凑摘要——块内各注意力头 key 向量的均值。这些摘要常驻 GPU，打分时只碰摘要，不碰已卸载到 CPU 或磁盘的完整 KV 块，块的相关性得分就是 query 与各头摘要的内积之和。论文用 Top-k 而非固定阈值，让候选块数量变化时选择尺度保持稳定。\n\n**动态预算控制。** 系统分别估算单块的拷贝成本和重算成本：拷贝按数据量除以主机到设备带宽计算，系数 2 对应 key 和 value 两份；重算按块长度与隐藏维平方估算，稀疏路径上只通过注意力投影从检查点重建，不重跑 MLP 也不走完整前向。两个成本的比例映射到目标保留率，再套一个 Minimum Keep Floor 作为保守下界，防止预算在压力下收缩得太狠。\n\n**按需局部重算加成本感知回退。** Top-k 选出的块往往散落各处，逐块恢复带来碎片化开销，所以先把相邻块合并成连续 span，再比较全量恢复与稀疏恢复的成本。当稀疏路径不再明显便宜时就退回全量，这就是 Dense Fallback Threshold γ 的作用。\n\n## 一台笔记本上的实测数字\n\n环境很具体：一台消费级笔记本，RTX 3070 Ti 8GB，Ryzen 9 6900HX 移动版 CPU，16GB 主机内存，Windows 加 WSL，引擎是 FlexGen\u002FFlexLLMGen，模型是 facebook\u002Fopt-6.7b，batch size 4。\n\nDeepSpeed 和 HF Accelerate 在 1k token 之前就 OOM，Quest 风格方案在 1200 token 处 OOM，只有 FlexGen 和 SparPR 保持可运行。FlexGen 靠溢出到 CPU 和磁盘完成 1k 到 4k，但在 8k 处超时；SparPR 把 1k、2k、4k 的端到端延迟分别降低 31%、52%、63%，吞吐提升 1.4 倍、2.1 倍、2.7 倍，并在 44 分钟内跑完 8k 负载。\n\n保真度用 Needle 任务测：把一条目标事实插进《爱丽丝梦游仙境》节选的两个深度位置。50% 保留率下 SparPR 在 2k\u002F4k\u002F8k 分别是 .943\u002F.879\u002F.780，H2O 是 .492\u002F.506\u002F.511，StreamingLLM 是 .445\u002F.467\u002F.485。压到 10% 时差距更明显：SparPR 还有 .600\u002F.482\u002F.308，H2O 掉到 .12 左右，StreamingLLM 崩到 .02。论文解释：前者基于位置，needle 只有落在最近窗口里才活下来；H2O 与查询无关，命中率基本跟着保留率走。\n\n消融里两个结果值得留意。去掉动态预算后准确率反而更高（.963 对比完整版 .957），说明预算机制的收益主要体现在更长上下文上。Quest-only 变体准确率只有 .777，是所有变体最低的——省下的延迟是拿检索质量换的。论文对局限也说得清楚：只有一个 OPT-6.7B、一块消费级 GPU、一个构造的 needle 任务。\n\n## 所以呢\n\n这篇论文的价值可能不在那几个漂亮的百分比，而在于它给出了一个更务实的框架：端侧推理的瓶颈管理，本质是不断在「搬」和「算」之间做动态权衡，而不是找到一个一劳永逸的压缩率。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.18140","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"efb2507b-8916-47d0-b05d-7885f58c2ad1","en","SparPR: Runtime KV-Cache Recovery on Consumer GPUs","SparPR splits the consumer-GPU KV cache bottleneck into three online decisions. No weight changes: 1k\u002F2k\u002F4k latency falls 31%\u002F52%\u002F63% on one laptop.","When you run a large language model on a consumer GPU, the real ceiling is often not the weights — it is the KV cache.\n\nWeights are loaded once, but the KV cache grows with every decoded token. Datacenter GPUs absorb that cost with large memory and high bandwidth. Swap in a laptop with 8GB of VRAM and the same cost shows up directly as throughput collapse, latency spikes, and OOM failures that arrive long before the model's nominal context window.\n\nA paper posted on arXiv on October 13, 2026 (SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware, https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.18140) takes a shot at this old problem. It does no compression and no quantization. Instead it splits \"the cache does not fit\" into three runtime decisions that can be made online.\n\n## Three decisions rather than one compression ratio\n\nThe paper sorts prior work into three disconnected lines: layered-offload systems such as FlexGen and LeoAM answer \"can it run at all\"; cache-retention methods such as StreamingLLM and H2O answer \"which information to keep\"; local-recovery methods such as Quest and ProphetKV answer \"how to get it back\". SparPR's claim is that these three questions belong in a single runtime loop.\n\n**Query-Aware key-block selection.** Each historical block gets a compact precomputed summary — the mean key vector across attention heads within that block. These summaries stay resident on the GPU, so scoring touches only the summaries and never the full KV blocks offloaded to CPU or disk. A block's relevance score is the sum of inner products between the query and each head's summary. The paper uses Top-k rather than a fixed threshold so that the selection scale stays stable as the candidate block count changes.\n\n**Dynamic budget control.** The system estimates the copy cost and the recompute cost of a single block separately. Copy cost is computed from data volume divided by host-to-device bandwidth, with a factor of 2 accounting for both key and value; recompute cost scales with block length and the square of the hidden dimension, and on the sparse path the KV states are rebuilt from a checkpoint through the attention projection alone — no MLP rerun, no full forward pass. The ratio of the two costs maps to a target keep ratio, which is then bounded by a Minimum Keep Floor so the budget never collapses too aggressively under pressure.\n\n**On-demand local recomputation with cost-aware fallback.** The blocks picked by Top-k are usually scattered across the context, and recovering them one by one brings fragmentation overhead. SparPR therefore merges adjacent blocks into continuous spans, then compares the cost of full recovery against sparse recovery. Once the sparse path is no longer clearly cheaper, the system falls back to the dense path — that is what the Dense Fallback Threshold γ governs.\n\n## Measured numbers on one laptop\n\nThe setup is deliberately concrete: a consumer laptop with an RTX 3070 Ti 8GB, an AMD Ryzen 9 6900HX mobile CPU, 16GB of host memory, Windows with WSL, FlexGen\u002FFlexLLMGen as the execution engine, and facebook\u002Fopt-6.7b at batch size 4.\n\nDeepSpeed and HF Accelerate run out of memory before 1k tokens, and a Quest-style query-aware scheme OOMs at 1200 tokens. Only FlexGen and SparPR stay runnable. FlexGen completes 1k through 4k by spilling to CPU and disk but times out at 8k; SparPR cuts end-to-end latency by 31%, 52%, and 63% at 1k, 2k, and 4k tokens, raises throughput by 1.4×, 2.1×, and 2.7×, and finishes an 8k-token workload in 44 minutes.\n\nFidelity is measured with a needle task: a target fact is inserted at two depth positions in an excerpt of Alice in Wonderland. At a 50% keep ratio SparPR scores .943\u002F.879\u002F.780 at 2k\u002F4k\u002F8k, while H2O scores .492\u002F.506\u002F.511 and StreamingLLM scores .445\u002F.467\u002F.485. At 10% the gap widens sharply: SparPR holds .600\u002F.482\u002F.308, H2O drops to around .12, and StreamingLLM collapses to .02. The paper's explanation: StreamingLLM is position-based, so a needle survives only if it lands in the recent window; H2O is query-agnostic, so its hit rate roughly tracks the keep ratio.\n\nTwo ablation results are worth noting. Removing dynamic budget actually raises accuracy (.963 versus .957 for the full version), which suggests the budget mechanism mainly pays off at longer contexts. The Quest-only variant reaches just .777, the lowest of any variant — the latency it saves is bought with retrieval quality. The paper is candid about the limits: a single OPT-6.7B model, one consumer GPU, and one constructed needle task.\n\n## So what\n\nThe value of this paper may not lie in the headline percentages but in a more pragmatic framing: bottleneck management for on-device inference is fundamentally a running trade-off between moving data and computing it, rather than the search for a one-and-done compression ratio.","sparpr-kv-cache-recovery-consumer-gpu","2026-10-22T01:20:00Z","2026-10-22T01:12:43.145369Z","2026-10-22T01:12:43.145381Z",true,"agent",42,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"5c1d994e-4baf-41ae-a57d-33411a8981b5","KAIST KV4-RL:INT4量化KV缓存,RL训练rollout提速1.8倍","kv4-rl-int4-kv-cache-rlvr-rollout","2026-10-15T19:11:53+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"3cecce90-70b9-4bb3-b9b7-93e6b0c05105","D-Quant 用熵编码压 KV:2.26bit 近无损","d-quant-entropy-coding-kv-cache","2026-09-20T17:10:42+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"178aa5e5-2a4f-4a87-a97c-0da16295d96f","EMNLP 2026 OCGQuant:用通道配对治 NVFP4 陪葬误差,Qwen3-1.7B 接近 FP16","ocgquant-nvfp4-outlier-companion-grouping","2026-09-10T09:15:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"4147f71b-eaa7-4d39-91cf-c2c572105e7f","FlashPrefill V2:128K 长文本 prefill 提速 47 倍,块稀疏注意力走进生产框架","flashprefill-v2-block-sparse-prefill","2026-08-21T19:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"c32d3160-4e07-4128-890f-4e135aac2cce","CompactifAI 把 Llama 3.3 70B 砍到一半:Multiverse 在 Intel Xeon 6 上跑出 1.9 倍吞吐","compactifai-llama-3-3-70b-intel-xeon","2026-07-26T04:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"cf7f8e7e-4efc-451e-9595-706b0be911ba","PolyQ:把\"3-bit LLM 跑在 CPU\"做成一件可预测的事","polyq-3bit-llm-cpu","2026-07-17T10:00:00+00:00"]