The bottleneck of LLM inference is shifting from compute to memory bandwidth: retrieval-augmented generation, inference-time compute scaling, and long-context applications push requests toward ever longer sequences, while memory capacity and communication bandwidth fail to keep pace — the well-known "memory wall." Autoregressive decoding must re-read the growing KV cache for every token generated, so data movement itself becomes the dominant cost in both latency and energy.

A team from UC Berkeley, ICSI, and LBNL submitted a paper titled "LLM Inference in a Flash!" to arXiv on September 14, proposing to move inference onto Flash compute-in-memory (Compute-in-Flash) devices, with algorithms tailored to that hardware. Authors include Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami (arxiv.org/abs/2609.16161).

Two Bottlenecks, Two Algorithms

Flash compute-in-memory moves computation next to the storage array, exploiting SSD capacity and high internal read bandwidth to bypass the memory wall. But running LLMs directly hits two obstacles: these devices lack support for high-precision floating-point operations, and their write endurance is limited — while the KV cache is exactly a high-frequency dynamic write workload, landing squarely on that weakness.

The team's response is two parallel algorithm tracks. The first is end-to-end integer-only quantization: from linear layers to nonlinear operators such as Softmax, RMSNorm, and SiLU, everything runs in integer approximations (polynomial/shift-based), eliminating floating-point computation entirely. On Llama-3.1-8B, the full integer pipeline achieves a WikiText-2 perplexity of 7.5803 versus an FP16 baseline of 7.5454 — a gap of just 0.0349; the paper notes that naive integer nonlinear approximations without INT16 requantization make activations diverge into NaN.

The second is dictionary-based KV cache compression built on sparse dictionary coding: a separate overcomplete dictionary of 32,768 atoms is trained per layer for keys and values, resident read-only in Flash; each KV vector becomes a linear combination of a few dictionary atoms (sparse codes), slashing write traffic and sidestepping the endurance limit. Combined with a 128-token local window and query-aware two-level sparsity (3 atoms for coarse scoring, refined to 16 for the top 10% of tokens), Llama-3.1-8B scores 43.65 average on LongBench against a 44.07 baseline while compressing dynamic KV cache traffic by 15×; Qwen-2.5-7B holds 44.23 versus 44.44, likewise retaining 99% of the score.

Why This Design Loves Flash

One honest discussion in the paper stands out: the same dictionary compression is not worthwhile on GPUs — the dictionary must be streamed repeatedly from HBM, making it slightly slower than no compression at 1K context (0.96×). Benefits only materialize when the dictionary stays resident in the CIM array and projections/gathers execute near the data. System-level modeling shows that at 256K long context, the CIM-SSD design delivers 4.4× latency and 6.8× energy savings versus an NPU+DRAM baseline; even at 1K short context, it achieves 3.1× latency and 2.7× energy gains.

So What

The real insight of this work is not the "15× compression" headline but the paradigm of algorithm-hardware co-design: instead of squeezing water out of the GPU memory hierarchy, reshape inference algorithms to match the physical constraints of novel storage hardware — integer execution bends to compute constraints, read-only dictionaries bend to write endurance. As HBM supply tightens and edge-side long-context demand grows, this "rewrite inference for Flash" route deserves attention. Admittedly, the paper also candidly notes its limits: it covers only the decode phase, and mapping the compute-bound prefill to CIM remains an open question.