arXiv 2603.12201 introduces IndexCache, a method that uses "cross-layer index reuse" to reduce the memory pressure of sparse attention. The result: 30B model prefill is 1.82× faster, with no quality loss.
The "sparse attention memory" problem: sparse attention methods (e.g., DSA, Sliding Window) reduce the FLOPs of attention, but they still require storing the full KV cache in memory. For 30B+ models with 1M context, the KV cache is hundreds of GB, and the memory bandwidth becomes the bottleneck.
The "cross-layer index reuse" insight: IndexCache observes that different attention layers tend to select the "same" sparse KV entries. If layer 5 selects entries {10, 20, 30}, then layer 6 is likely to also select entries in this set. The "index" of the selected entries can be reused across layers, with only the "values" needing to be unique per layer.
The technical details: IndexCache stores the "index" of selected entries once, and each layer's "value" is stored separately. The cross-layer index is computed once per forward pass and reused. The result is 2-3× memory bandwidth reduction, which translates directly to faster prefill.
The benchmark: on a 30B model with 1M context, IndexCache-1.82× prefill speedup, with quality loss of less than 0.3 points. The speedup is most significant on long-context tasks (where the memory bandwidth is the bottleneck), and the quality is preserved because the index is exact (no approximation).
The bigger takeaway: "cross-layer reuse" is a significant new direction for sparse attention. The "each layer is independent" assumption is wasteful, and the "cross-layer reuse" approach is significantly more efficient. For the industry, this signals that "sparse attention" will move to cross-layer designs, and the next round of inference improvements will come from "how much cross-layer structure is exploited."