[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tensor-parallelism-vs-kv-compression-cost":3,"topics-all":38,"news-related-d31bc388-b6c7-41a5-a6e9-6f00657c7616":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"d31bc388-b6c7-41a5-a6e9-6f00657c7616","加GPU还是压KV缓存？arXiv论文：压缩省钱1.2到2倍，但36B是道坎","LLM推理显存吃紧时有两条主流出路：加GPU张量并行，或压缩KV缓存。arXiv 2608.23962在A100\u002FA40\u002FH100校准的模拟器上把两者放到同一成本轴：没找到交叉点，压缩便宜1.2到2倍；80GB卡约36B参数是分水岭，之上张量并行只是入场券。","LLM 推理服务跑着跑着显存不够了，工程师面前通常摆着两条路：要么加 GPU，把权重和 KV 缓存切分到多张卡上；要么把 KV 缓存原地压小，单卡继续跑，拿一点质量换空间。这两条路来自两个几乎不对话的社区——系统社区管扩容，算法社区管压缩，各自汇报各自的指标：压缩论文讲内存比例，扩容论文讲吞吐曲线，很少被放到同一张成本账单上比较。arXiv 8 月 25 日提交的新论文《More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving》补上了这个空白（[论文地址](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.23962)）。\n\n## 实验设计：把两条路线拉上同一根成本轴\n\n作者用了一个经过性能剖析、在 A100、A40、H100 三种真实硬件上校准的模拟器，把张量并行配置（并行度 1 到 8）和 KV 压缩配置（16\u002F8\u002F4-bit 量化、保留率最低压到 0.25）放进同一坐标系：纵轴是每百万 token 成本，横轴是延迟。测试对象是 Llama-2 的 7B 和 70B 两档模型。\n\n论文还点破了一个业界最常见的对比错误：算成本时省掉并行度 p，等于假装扩容不要钱。作者的原话是，省略 p 是他们见过最容易出错的地方，去掉 p 会推翻本文所有结论——这也是很多「加卡更快」的直觉经不起推敲的原因。\n\n## 核心结论：交叉点不存在\n\n作者原本想找「成本等效交叉点」——某个规模之上加卡反超压缩——结果是没找到。在他们构造的每一种显存压力水平下，压缩都比加卡便宜 1.20 到 2.00 倍，且显存压力越深差距越大。\n\n更重要的是分水岭：对 80GB 单卡，大约 36B 参数是策略边界。低于这道墙，压缩全面占优，多买的 GPU 大概率是浪费的钱——论文甚至指出 7B 模型在 80GB 设备上根本无法在自己的上下文窗口内耗尽 KV 预算；高于这道墙，张量并行不再是选项而是入场券：Llama-2-70B 在单张 A100 上无论怎么设置 KV 都跑不起来，因为卡住的是权重本身，而 KV 压缩动不了权重。\n\n两个杠杆各有不可替代的一侧：张量并行是唯一能改善延迟的杠杆，压缩反而让每 token 延迟变差 8% 到 93%（来自 batching 争用）；压缩则是唯一能放大单位美元容量的杠杆——16.5 倍，而 8 倍的 GPU 开销只换来 1.21 倍。\n\n## 所以呢\n\n这篇论文的价值不在「永远选压缩」这样的口号，而在给出一套决策次序：先看模型权重相对单卡显存的比例——塞得进单卡，买卡扩容前先试压缩；塞不进去，讨论「要不要张量并行」本身就是伪问题。再问自己在优化延迟还是容量，因为这两个目标分别指向两根不同的杠杆。对正在做推理成本核算的团队，这篇账单值得放进收藏夹。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.23962","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"ec61fd07-f67f-439a-8148-89fc8df02944","en","More GPUs or a Smaller KV Cache? arXiv Paper: Compression Is 1.2-2x Cheaper, With a 36B Wall","When LLM serving runs out of KV cache room, teams either add GPUs via tensor parallelism or compress the KV cache in place. arXiv 2608.23962 puts both on one cost-normalised axis using a simulator calibrated on A100\u002FA40\u002FH100: no crossover exists, compression is 1.20-2.00x cheaper, and roughly 36B parameters marks the strategy boundary on an 80GB card.","When an LLM serving deployment runs out of KV cache room, engineers face two established escape routes: add GPUs and shard the weights and KV cache across them via tensor parallelism, or shrink the cache in place with KV quantisation and eviction while keeping a single GPU. The two routes come from two communities that rarely talk to each other - systems people handle scaling out, algorithms people handle compression - and each reports its own metrics: compression papers report memory ratios, parallel-scaling papers report throughput curves, and almost nobody puts the two on the same cost bill. A new paper submitted to arXiv on August 25, \"More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving\", fills that gap ([paper link](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.23962)).\n\n## The Setup: One Cost-Normalised Axis\n\nThe authors use a profiled simulator calibrated on real A100, A40, and H100 hardware, placing tensor-parallel configurations (degree 1 to 8) and KV-compressed configurations (16\u002F8\u002F4-bit quantisation, keep-ratios down to 0.25) into one coordinate system: cost per million tokens against latency. The test subjects are Llama-2 at 7B and 70B.\n\nThe paper also calls out the most common mistake in this comparison: omitting the parallel degree p when computing cost, which amounts to pretending scale-out is free. In the authors words, dropping p would reverse every conclusion in the paper - which is exactly why the intuition that \"more cards means faster and better\" often does not survive scrutiny.\n\n## The Finding: No Crossover Exists\n\nThe authors went looking for a cost-equivalence crossover - a scale above which adding GPUs beats compression - and did not find one. Across every level of memory relief they constructed, compression is cheaper by 1.20x to 2.00x, and the gap widens as relief deepens.\n\nThe more important discovery is the boundary: for an 80GB card, roughly 36B parameters is the dividing line. Below that wall, compression dominates and extra GPUs are largely wasted spend - the paper notes a 7B model on an 80GB device cannot exhaust its KV budget within its own context window. Above it, tensor parallelism stops being a choice and becomes an entry ticket: Llama-2-70B is infeasible on a single A100 at any KV setting, because the binding resource is the weights themselves, which KV compression does not touch.\n\nEach lever owns an irreplaceable side: tensor parallelism is the only lever that improves latency, while compression makes per-token latency worse by 8% to 93% through batching contention; compression is the only lever that multiplies capacity per dollar - 16.5x, against 1.21x for an eightfold spend on GPUs.\n\n## So What\n\nThe value of this paper is not a slogan like \"always compress\", but a decision order: first check the ratio of model weights to single-card memory - if it fits on one card, try compression before buying cards; if it does not fit, debating \"whether to use tensor parallelism\" is a pseudo-question. Then ask whether you are optimising latency or capacity, because those two goals point at two different levers. For teams doing inference cost accounting, this bill belongs in your bookmarks.","tensor-parallelism-vs-kv-compression-cost","2026-08-30T17:10:00Z","2026-08-30T17:10:45.644958Z","2026-08-30T17:10:45.644970Z",true,"agent",215,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"c83af54b-79ed-445c-9482-07d98c26c36b","BeaconKV:长推理会回头看,只压最近窗口的 KV 缓存注定丢东西","beaconkv-beacon-query-kv-cache-compression","2026-09-09T11:25:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"bce9fc16-d31a-49be-b17b-f144619a58e2","LatentPress:上下文压成软令牌直读，7.7 倍压缩反超原文，训练仅动 0.1% 参数","latentpress-soft-token-context-compression","2026-09-05T19:06:09+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"d2c56430-f9a7-4844-bfab-a8651279a70c","ResKV 不再把 KV 缓存压缩等同于删词：给被淘汰的信息留一份残差账本","reskv-residual-kv-cache-compression","2026-08-03T10:43:23+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"c32d3160-4e07-4128-890f-4e135aac2cce","CompactifAI 把 Llama 3.3 70B 砍到一半:Multiverse 在 Intel Xeon 6 上跑出 1.9 倍吞吐","compactifai-llama-3-3-70b-intel-xeon","2026-07-26T04:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"5f745fe5-ea5d-453a-8b08-7dac524d1ac2","ACL 2026 综述 sKis：KV 缓存优化重塑为 LLM serving 系统学","acl-2026-skis-kv-cache","2026-07-12T18:15:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"dd9d199c-5cd9-4a15-8ae6-7e4fb40f4129","MosaicKV:把 KV 缓存压成「马赛克」,长上下文推理跑出 16× 注意力加速","mosaickv-mosaic-compression","2026-07-03T18:01:00+00:00"]