Turning the sampling temperature up can make a model explore more, and long generations can start to fall apart — a scene that would not surprise many debugging sessions. What makes this paper interesting is not the model but the gap between the model and the serving engine: a memory saving promised in a technical report often never shows up in a real deployment.

37.5% in the report, 1024 in the engine

Gemma 4's 31B model has two layer types: 50 sliding-window layers attend to the last 1,024 tokens with 16 KV heads of 256 dimensions, and 10 global layers attend to the whole context with 4 KV heads of 512 dimensions. At long context the global layers hold most of the cache of a request. Their model configuration calls the structure attention_k_eq_v, and they have no value projection: the value is the RMS-normalized key projection, and the key is that value multiplied by a fixed weight, with rotary position encoding on 128 of the 512 dimensions. The Gemma 4 technical report states that this design, combined with KV cache sharing, reduces the global KV cache by up to 37.5% (arXiv:2607.02770).

The paper read the KV-cache code of vLLM, SGLang, TensorRT-LLM and Hugging Face transformers, and searched 15 repositories plus arXiv and the web. Its finding is that these layers still store 1,024 values per KV head and token, so the promised saving is not realized in the engines. The paper adds a caveat of its own: a search cannot see private code. The paper is at https://arxiv.org/abs/2611.23035

640 numbers doing the work of 1024

The proposal is a single row of 640 values per KV head and token, stored in three pieces: 128 values of k_R, 384 of v_U and 128 of v_R. The first 512 values are the key that scoring needs, the last 512 are the value, and the middle 384 overlap. The point is that these are not two copies: they are two strided views over the same buffer, so nothing is copied and an unchanged paged attention kernel reads them as they are.

Keeping the kernel unchanged works because the missing part of the key is folded into the query, and rotary position encoding does not touch that part, so in real arithmetic the fold is exact. In 16-bit arithmetic it is not bit-exact — the two layouts round differently, and the permuted dimension order changes the order of the floating-point sums. What is bit-identical is the compact build against the emulation: the latter computes the same arithmetic on the stock cache, and the outputs matched bit for bit on all 432 prompts.

Why not store only 512 values? The paper gives the reason: the rotation angle differs for every cached token, so it cannot be applied once to the query, and a 512-value row would need a new kernel that rotates per token while reading the cache. Storing 640 trades a little space for a layout that needs no new kernel. The paper states plainly that it does not claim the saving, the shape of the stored row or the exact cache reduction as new.

Memory goes down, speed does not follow

Measured in vLLM with the 16-bit 31B model on one A100 and a 15 GiB cache pool: at 32K context the steady decode footprint of one request fell from 1,349 blocks to 1,008, and at 64K from 2,373 to 1,690, which is 0.83 and 1.67 GiB less KV memory per request; the global blocks fell to about two thirds. At 32K the same pool admitted one more request, and the decode rate rose from 56.0 to 69.7 tokens per second. At 64K both layouts held two requests, and the compact build was 4% slower.

On accuracy, the 432 items were 400 GSM8K test problems and 32 needle-retrieval prompts of 7.7K tokens; the compact build scored 0.9769 against the stock engine's 0.9722, a 0.46-point difference inside the preset margin, and the two changed final answers both moved toward the correct answer, with the generated text differing on 88 of the 432 items.

The gain shows up where prefill and decode are separated. On a decode node built from two A100s, the compact engine ran 1.376 times the stock decode tokens per work step at a 16K median prompt and 1.237 times at 8K, against a block-pool model frozen before the runs that predicted 1.364 and 1.229. Eager decode throughput rose by 1.30 and 1.11 times. The cost is written down too: a compact step took 5.5% longer at the longer prompt and 11% longer at the shorter one, and the run-time prototype costs 4% to 9% of an eager step at equal batch.

The cold water

The paper's own limits are worth keeping in view. The accuracy test covers prompts up to 7.7K tokens, while the lengths where the saving actually matters are longer, and the author's wording is 'no break found', not proof of equal quality. On the FP8 path, at the default KV scale the compact row fails the paper's attention-error bound, so the 1.34 to 1.40 factor applies only to a 16-bit cache. Every measurement covers the 31B model and one vLLM build, with no test of prefix caching, tensor parallelism or speculative decoding. Most importantly, the prototype code and the experiment records are described as planned releases, and they are not out yet.

So what

One account is worth keeping straight. The paper says it read 16 current model configurations, and that other exact redundancies of 10% or more that a model declares in its KV cache were already used by vLLM or by the serving image that the model supports. The equal key and value design in Gemma 4 is a different case: there is no matching engine-side implementation, and that is why the 37.5% is still on the table.

The more valuable account is this: the saving stated in a model report is a note payable to the serving engine. The value of this paper is not a new idea but a precise accounting of that note — how the 37.5% slipped through the engines, how much each step costs once the saving lands, and under which conditions the account turns into throughput rather than a number on paper. Once an engine actually adopts the layout, the next question arrives: who should price notes like this, the model team or the inference framework team?