IBM Research released llm-d, a new inference framework optimized for mixed-vendor GPU clusters. The standout: 3-5× speedup over single-vendor inference, achieved through KV-cache-aware routing that can place different layers of a model on different vendors' GPUs.
The technical details: llm-d is built on top of vLLM, with a "heterogeneous KV cache" extension that allows the KV cache to be split across different vendors' GPUs. A "cache-aware router" decides which GPU to place each layer on, based on the current cache hit rate, the GPU's memory bandwidth, and the model's layer-level latency profile. The router is dynamic — it can re-route layers during inference if the load changes.
The benchmark: on a cluster of 64 GPUs (mixing NVIDIA H100, AMD MI300X, and Intel Gaudi 3), llm-d serves IBM Granite-70B at 3.2× the tokens-per-second of the best single-vendor configuration, and serves Sarvam-30B at 4.7×. The framework is open-sourced and supports any model that fits vLLM.
The strategic angle: "mixed-vendor GPU" is becoming a real deployment scenario. Most enterprises don't want to be locked into a single GPU vendor, and the supply constraints of NVIDIA have made multi-vendor procurement a necessity. llm-d is the first framework to make this practical — without it, mixed-vendor clusters have 40-60% efficiency loss.
The bigger takeaway: "GPU vendor lock-in" is being broken. The "all NVIDIA" assumption that dominated the first 3 years of LLM inference is ending, and frameworks like llm-d are enabling a more diverse GPU ecosystem. For the industry, this means AMD and Intel can finally compete on inference workloads, and the NVIDIA premium will likely come down.