On July 1, Korean AI-infrastructure company Dnotitia, together with UC San Diego's VVIP Lab, open-sourced STAR-KV, which has been accepted as an ICML 2026 Spotlight (roughly 8.4% of accepted papers). On the LLaMA-3.1-8B setup with 128K context and batch size 4, the KV cache eats around 81% of GPU memory — STAR-KV sets out to directly answer: "Can the long-context inference memory wall be compressed to the point where it's no longer the bottleneck?"

Three moves chained into one pipeline

The full name of STAR-KV is "Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control." The technique packages three common ideas into one pipeline:

  1. Low-rank approximation first slashes the KV by 75%; 2. Mixed-precision quantization pushes the compression ratio up to 20×; 3. A custom GPU kernel reduces redundant attention memory accesses.

The authors report: attention compute speedups up to 6.9×, generation throughput up to 3.1×, and accuracy on multiple evaluations higher than KVTC (ICLR 2026, Transform Coding) and Google's TurboQuant. The code is open-sourced, and integration into vLLM is the next milestone.

The real differentiator is "soft thresholding"

The KV-compression line — from H2O, to KVTC, to TurboQuant — has already gone through several rounds. STAR-KV's differentiator is "soft thresholding + adaptive rank": the model decides the low-rank approximation level per layer and per context segment, instead of hard-cutting with a single set of Transform Coding coefficients as KVTC does. STAR-KV turns "how much to compress" from a hand-tuned hyperparameter into a learnable control variable — finer granularity, more controllable accuracy loss.

Engineering is taking over the field

The ICML 2026 main conference is in Seoul, and Dnotitia — a Korean AI-infra company — landing the paper in Spotlight, combined with the open-source release and the planned vLLM integration, signals a turning point: the long-context inference track has moved from "papers racing on speedup ratios" into the "engineering-selection" stage.

For the Chinese community, beyond the headline numbers, the more meaningful metric to watch is: who first gets end-to-end long-context throughput running stably on vLLM / SGLang while keeping accuracy intact. With TurboQuant and STAR-KV arriving in quick succession, the notion that "long context = expensive" may be approaching its engineering inflection point — provided these compression methods can actually hold up under long-horizon agent and 1M-token-context production loads.