LLM serving retires heuristics: the operations-research lens

LLM inference services have reached tens of billions of requests per day, but core algorithms still rely on the heuristic strategies of general-purpose distributed systems — shortest-queue routing, FIFO scheduling, LRU caching. An arXiv paper published on May 2 points out that LLM inference's structural uniqueness (dynamically growing KV Cache, prefill-decode phase asymmetry, unknown output length, continuous batching) means general heuristics miss substantial optimization space. The paper argues that the intersection of operations research and ML systems has proven that principled methods can match or exceed heuristic performance while providing theoretical guarantees.

Specifically, the core algorithms of current LLM serving systems (like vLLM and SGLang) have barely changed: request routing is still shortest-queue or round-robin, scheduling defaults to FIFO, KV cache reclamation uses LRU. These general strategies completely ignore LLM inference's special structure. The paper advocates that mathematical models capable of capturing these features must be developed for LLM serving, with algorithms that have provable performance guarantees, rather than heuristics that may work in some scenarios but fail unpredictably in others.

This problem is especially acute in MoE load-balancing scenarios: when tokens concentrate on a few hot experts, the GPUs hosting those experts become the bottleneck, while other GPUs can only sit idle. Current main balancing strategies use auxiliary loss functions to penalize uneven token distribution across experts, but this introduces gradient interference conflicting with the main language-modeling objective. The paper argues more principled methods are needed to handle these issues.

From an industry perspective, the paper's value is that it's not empty theoretical talk — it explicitly identifies specific decision problems in LLM serving (request routing, scheduling, cache management, load balancing, capacity planning, resource allocation), all of which are amenable to formal analysis. As inference engine architectures like vLLM and SGLang stabilize, algorithm-level innovation becomes a durable investment, not requiring redesign with every system incremental update.