When an LLM serving deployment runs out of KV cache room, engineers face two established escape routes: add GPUs and shard the weights and KV cache across them via tensor parallelism, or shrink the cache in place with KV quantisation and eviction while keeping a single GPU. The two routes come from two communities that rarely talk to each other - systems people handle scaling out, algorithms people handle compression - and each reports its own metrics: compression papers report memory ratios, parallel-scaling papers report throughput curves, and almost nobody puts the two on the same cost bill. A new paper submitted to arXiv on August 25, "More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving", fills that gap (paper link).
The Setup: One Cost-Normalised Axis
The authors use a profiled simulator calibrated on real A100, A40, and H100 hardware, placing tensor-parallel configurations (degree 1 to 8) and KV-compressed configurations (16/8/4-bit quantisation, keep-ratios down to 0.25) into one coordinate system: cost per million tokens against latency. The test subjects are Llama-2 at 7B and 70B.
The paper also calls out the most common mistake in this comparison: omitting the parallel degree p when computing cost, which amounts to pretending scale-out is free. In the authors words, dropping p would reverse every conclusion in the paper - which is exactly why the intuition that "more cards means faster and better" often does not survive scrutiny.
The Finding: No Crossover Exists
The authors went looking for a cost-equivalence crossover - a scale above which adding GPUs beats compression - and did not find one. Across every level of memory relief they constructed, compression is cheaper by 1.20x to 2.00x, and the gap widens as relief deepens.
The more important discovery is the boundary: for an 80GB card, roughly 36B parameters is the dividing line. Below that wall, compression dominates and extra GPUs are largely wasted spend - the paper notes a 7B model on an 80GB device cannot exhaust its KV budget within its own context window. Above it, tensor parallelism stops being a choice and becomes an entry ticket: Llama-2-70B is infeasible on a single A100 at any KV setting, because the binding resource is the weights themselves, which KV compression does not touch.
Each lever owns an irreplaceable side: tensor parallelism is the only lever that improves latency, while compression makes per-token latency worse by 8% to 93% through batching contention; compression is the only lever that multiplies capacity per dollar - 16.5x, against 1.21x for an eightfold spend on GPUs.
So What
The value of this paper is not a slogan like "always compress", but a decision order: first check the ratio of model weights to single-card memory - if it fits on one card, try compression before buying cards; if it does not fit, debating "whether to use tensor parallelism" is a pseudo-question. Then ask whether you are optimising latency or capacity, because those two goals point at two different levers. For teams doing inference cost accounting, this bill belongs in your bookmarks.