[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-gemma-4-kv-cache-k-equals-v-compact-row":3,"topics-all":38,"news-related-9af27b13-b1a9-47e3-a143-fd8a6c1ee5a9":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"9af27b13-b1a9-47e3-a143-fd8a6c1ee5a9","Gemma 4省下的37.5%,推理引擎还没兑现","Gemma 4 技术报告说全局层复用键值,全局 KV cache 最多省 37.5%,但论文查过 vLLM 等引擎后发现每个 KV 头每个 token 仍存满 1024 个值。640 个数的新布局靠两个带步长的视图被未改动的内核读出,每请求解码 KV 内存降到 1.34 到 1.40 分之一。","把温度调高能让模型多探索一点,长文本生成却开始胡言乱语——这类场景在很多团队的调试现场不算新闻。这篇论文有意思的地方不在模型本身,而在模型和推理引擎之间那道一直没人补的缝:一份技术报告承诺的内存节省,在真实部署里往往一笔都没兑现。\n\n## 报告里的 37.5%,引擎里的 1024\n\nGemma 4 的 31B 模型有两类层:50 个滑动窗口层只看最近 1024 个 token,配 16 个 KV 头、每头 256 维;10 个全局层看整个上下文,配 4 个 KV 头、每头 512 维。长上下文下,全局层才是每请求缓存的大头。这批层的模型配置项叫 attention_k_eq_v,没有独立的 value 投影:value 就是 RMS 归一化后的 key 投影,key 则是这个 value 乘上一个固定权重,位置编码只作用在 512 维里的 128 维。Gemma 4 技术报告的说法是,这套设计配上 KV 共享,全局 KV cache 最多省 37.5%(arXiv:2607.02770)。\n\n论文读了 vLLM、SGLang、TensorRT-LLM、Hugging Face transformers 的 KV cache 代码,又搜了 15 个仓库、arXiv 和网页,给出的判断是:每个 KV 头每个 token 仍然存满 1024 个值,那份节省没有被引擎兑现。论文自己补了一句,搜索看不到私有代码。这篇论文见 https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.23035\n\n## 640 个数干 1024 个数的活\n\n办法是一行 640 长度的行:每个 KV 头每个 token 存 k_R、v_U、v_R 三段,分别是 128、384、128 个数,合计 640。前 512 个数是打分需要的 key,后 512 个数是 value,中间 384 个是重叠部分。要点是这两段不是两份拷贝,而是同一块内存上的两个带步长的视图,不需要搬任何数据,现成的分页注意力内核照原样就能读。\n\n不改内核的关键在 query:key 里缺的那部分被折叠进 query,而位置编码不碰这一段,所以在实数运算下这一步是精确的。16 位下则不是逐位一致——论文说两种取整路径不同,维度置换也改了浮点求和顺序。真正逐位相同的是紧凑版与对照版:后者在 stock cache 上模拟同样的算术,432 个提示的输出全部逐位一致。\n\n为什么不干脆只存 512 个?论文给了理由:旋转角度对每个缓存 token 都不同,没法一次性作用在 query 上,那就得换一个新内核。存 640 个,是拿一点空间换一个不用改内核的方案。论文明确写着,不主张这个节省、这个行形状或者这个缓存缩减是新的。\n\n## 内存省下来了,速度不一定\n\nvLLM 里用 16 位 31B 模型、单张 A100、15 GiB 缓存池实测:32K 上下文下,每请求稳态解码占用从 1349 个块降到 1008 个,64K 下从 2373 降到 1690,折算成每请求少占 0.83 和 1.67 GiB;全局层的块数降到原来的三分之二。32K 时同一个池子能多塞进一个请求,解码速度从 56.0 涨到 69.7 token 每秒。但 64K 时两种布局都只装得下两个请求,紧凑版反而慢了 4%。\n\n精度那边,432 道题里有 400 道 GSM8K 和 32 道 7.7K token 的针检索,紧凑版 0.9769 对 stock 的 0.9722,差 0.46 分,落在预设的正负 3 分区间里;两道最终答案变了,都转向正确答案一侧,432 道题里 88 道的生成文本不同。\n\n收益真正出现的地方是预填充与解码分离的部署。两张 A100 组成的解码节点上,16K 提示长度下每步解码 token 数是原来的 1.376 倍,8K 下是 1.237 倍,论文事先冻结的块池模型分别预测了 1.364 和 1.229。eager 解码吞吐涨到 1.30 和 1.11 倍。代价也写清楚了:紧凑版每一步在长提示下慢 5.5%、短提示下慢 11%,单机测试里同批下一步慢 4% 到 9%。\n\n## 该泼的冷水\n\n论文的自我限定写得很诚实:精度测试只覆盖到 7.7K token 的提示,而省内存真正有意义的场景要长得多,作者的说法是「没发现破坏」,不是「证明质量相同」;FP8 路径上,默认 KV 精度下紧凑行过不了误差界,1.34 到 1.40 倍的节省只对 16 位成立;所有测量只覆盖 31B 一个模型和 vLLM 一个版本,没测前缀缓存、张量并行和投机解码;最要紧的是,原型代码和实验记录论文写的是「计划发布」,目前还没有。\n\n## 所以呢\n\n有一条账算得挺清楚。论文说它读了 16 份当前模型配置,别的模型声明的 10% 以上精确冗余,基本都已经被 vLLM 或者配套的推理镜像用掉了;Gemma 4 这个键值相等的设计属于另一类情况,引擎侧没有对应实现,这也是它能省 37.5% 的原因。\n\n另一条更值钱:模型报告里的省,本质是开给推理引擎的欠条。这篇论文的价值不在提出新东西,而在把欠条金额算到分:37.5% 是怎么在引擎里漏掉的,省下来之后每一步要慢多少,以及在什么条件下这笔账才变成吞吐而不是纸面数字。等推理引擎真把这套布局吃进去,下一个要回答的问题就来了:这类欠条,该由模型团队定价,还是由推理框架团队定价?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.23035","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3407704a-5855-4ac6-8673-3ae20c5df3d9","en","Gemma 4's 37.5% KV cache saving is not in the engines","Gemma 4 promises 37.5% less global KV cache, yet engines store 1,024 values per KV head. A 640-value row cuts decode KV memory 1.34 to 1.40 times.","Turning the sampling temperature up can make a model explore more, and long generations can start to fall apart — a scene that would not surprise many debugging sessions. What makes this paper interesting is not the model but the gap between the model and the serving engine: a memory saving promised in a technical report often never shows up in a real deployment.\n\n## 37.5% in the report, 1024 in the engine\n\nGemma 4's 31B model has two layer types: 50 sliding-window layers attend to the last 1,024 tokens with 16 KV heads of 256 dimensions, and 10 global layers attend to the whole context with 4 KV heads of 512 dimensions. At long context the global layers hold most of the cache of a request. Their model configuration calls the structure attention_k_eq_v, and they have no value projection: the value is the RMS-normalized key projection, and the key is that value multiplied by a fixed weight, with rotary position encoding on 128 of the 512 dimensions. The Gemma 4 technical report states that this design, combined with KV cache sharing, reduces the global KV cache by up to 37.5% (arXiv:2607.02770).\n\nThe paper read the KV-cache code of vLLM, SGLang, TensorRT-LLM and Hugging Face transformers, and searched 15 repositories plus arXiv and the web. Its finding is that these layers still store 1,024 values per KV head and token, so the promised saving is not realized in the engines. The paper adds a caveat of its own: a search cannot see private code. The paper is at https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.23035\n\n## 640 numbers doing the work of 1024\n\nThe proposal is a single row of 640 values per KV head and token, stored in three pieces: 128 values of k_R, 384 of v_U and 128 of v_R. The first 512 values are the key that scoring needs, the last 512 are the value, and the middle 384 overlap. The point is that these are not two copies: they are two strided views over the same buffer, so nothing is copied and an unchanged paged attention kernel reads them as they are.\n\nKeeping the kernel unchanged works because the missing part of the key is folded into the query, and rotary position encoding does not touch that part, so in real arithmetic the fold is exact. In 16-bit arithmetic it is not bit-exact — the two layouts round differently, and the permuted dimension order changes the order of the floating-point sums. What is bit-identical is the compact build against the emulation: the latter computes the same arithmetic on the stock cache, and the outputs matched bit for bit on all 432 prompts.\n\nWhy not store only 512 values? The paper gives the reason: the rotation angle differs for every cached token, so it cannot be applied once to the query, and a 512-value row would need a new kernel that rotates per token while reading the cache. Storing 640 trades a little space for a layout that needs no new kernel. The paper states plainly that it does not claim the saving, the shape of the stored row or the exact cache reduction as new.\n\n## Memory goes down, speed does not follow\n\nMeasured in vLLM with the 16-bit 31B model on one A100 and a 15 GiB cache pool: at 32K context the steady decode footprint of one request fell from 1,349 blocks to 1,008, and at 64K from 2,373 to 1,690, which is 0.83 and 1.67 GiB less KV memory per request; the global blocks fell to about two thirds. At 32K the same pool admitted one more request, and the decode rate rose from 56.0 to 69.7 tokens per second. At 64K both layouts held two requests, and the compact build was 4% slower.\n\nOn accuracy, the 432 items were 400 GSM8K test problems and 32 needle-retrieval prompts of 7.7K tokens; the compact build scored 0.9769 against the stock engine's 0.9722, a 0.46-point difference inside the preset margin, and the two changed final answers both moved toward the correct answer, with the generated text differing on 88 of the 432 items.\n\nThe gain shows up where prefill and decode are separated. On a decode node built from two A100s, the compact engine ran 1.376 times the stock decode tokens per work step at a 16K median prompt and 1.237 times at 8K, against a block-pool model frozen before the runs that predicted 1.364 and 1.229. Eager decode throughput rose by 1.30 and 1.11 times. The cost is written down too: a compact step took 5.5% longer at the longer prompt and 11% longer at the shorter one, and the run-time prototype costs 4% to 9% of an eager step at equal batch.\n\n## The cold water\n\nThe paper's own limits are worth keeping in view. The accuracy test covers prompts up to 7.7K tokens, while the lengths where the saving actually matters are longer, and the author's wording is 'no break found', not proof of equal quality. On the FP8 path, at the default KV scale the compact row fails the paper's attention-error bound, so the 1.34 to 1.40 factor applies only to a 16-bit cache. Every measurement covers the 31B model and one vLLM build, with no test of prefix caching, tensor parallelism or speculative decoding. Most importantly, the prototype code and the experiment records are described as planned releases, and they are not out yet.\n\n## So what\n\nOne account is worth keeping straight. The paper says it read 16 current model configurations, and that other exact redundancies of 10% or more that a model declares in its KV cache were already used by vLLM or by the serving image that the model supports. The equal key and value design in Gemma 4 is a different case: there is no matching engine-side implementation, and that is why the 37.5% is still on the table.\n\nThe more valuable account is this: the saving stated in a model report is a note payable to the serving engine. The value of this paper is not a new idea but a precise accounting of that note — how the 37.5% slipped through the engines, how much each step costs once the saving lands, and under which conditions the account turns into throughput rather than a number on paper. Once an engine actually adopts the layout, the next question arrives: who should price notes like this, the model team or the inference framework team?","gemma-4-kv-cache-k-equals-v-compact-row","2026-11-18T11:40:00Z","2026-11-18T11:17:17.740443Z","2026-11-18T11:17:17.740457Z",true,"agent",136,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"53d00d00-e4fb-41b0-a36a-37161203b514","KV缓存拼接复用丢题可至100%,修补要盯答案块","kv-cache-repair-budget-target-chunk","2026-11-05T15:09:02+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"02c96622-1409-4867-9f25-772936edc6cd","滑动窗口反超SnapKV:NVIDIA重审投机解码KV压缩","rift-draft-kv-cache-compression","2026-10-23T15:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a046b158-396f-4db9-a8d3-22d8c193a166","清华把KV缓存当神经网络训练:32倍压缩准确率96.7%","latent-compaction-test-time-kv","2026-10-15T17:08:40+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"dab2e487-cfba-4d75-a93f-eff133476348","剪枝校准对不上解码分布?西湖蚂蚁让稀疏解码提速48%","sparsedecoding-decoding-aware-pruning","2026-10-10T19:09:23+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"c83af54b-79ed-445c-9482-07d98c26c36b","BeaconKV:长推理会回头看,只压最近窗口的 KV 缓存注定丢东西","beaconkv-beacon-query-kv-cache-compression","2026-09-09T11:25:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"bce9fc16-d31a-49be-b17b-f144619a58e2","LatentPress:上下文压成软令牌直读，7.7 倍压缩反超原文，训练仅动 0.1% 参数","latentpress-soft-token-context-compression","2026-09-05T19:06:09+00:00"]