LLM serving systems hide a row of knobs nobody wants to touch: how much KV cache each layer retains, how many experts each MoE layer activates. The industry default is a uniform policy or a heuristic borrowed from some compression paper — one global setting, shipped. A Huawei team's InferOpt paper (arXiv 2610.04473, submitted October 3) argues for a different route: stop inventing the N+1th compression heuristic and treat inference configuration as what it really is — constrained multi-objective optimization.
Turning layer-wise knobs into a search problem
Why is this hard? In KV-cache compression, Qwen2.5-7B has one retention ratio per layer — a 28-dimensional continuous space. MoE routing is worse: DeepSeek-V2-Lite's per-layer activated-expert count ranges from 1 to 6, a 26-dimensional discrete space. Evaluating a single configuration on the full benchmark can take hours on a billion-parameter model, so exhaustive search is off the table. InferOpt runs three stages: it freezes a sampled proxy subset so every candidate is scored on identical examples; a closed-form resource-cost function rejects over-budget candidates before any model call; the budget tightens adaptively as feasible solutions accumulate, and Pareto-front representatives are re-validated on the full dataset. Plugging in a new mechanism requires only variable bounds, a resource-cost function, and an evaluation hook — the search pipeline itself is untouched.
The numbers: 64% less KV cache, 43% fewer routed pairs
On the KV side, at an aggressive mean-retention budget of 0.321, the 16K-context cache drops from 902.9 MiB to about 320 MiB — a 64.4% cut — while TPOT falls from 19.11ms to 9.83ms, a 22.9–48.5% reduction. The controlled comparison is the interesting part: at matched budgets, the searched layer-wise allocation beats uniform allocation by 3.21 to 6.13 quality points (7.3% and 14.0%), while swapping the selection scorer (attention-mass vs key-L2) moves quality by only 0.52 points. Allocation matters more than selection — the paper's most counterintuitive finding. On the MoE side, the searched segmented schedule (k=6 for the first 9 layers, k=3 for the middle 9, k=1 for the final 8) removes 43.0% of routed token–expert pairs and cuts TPOT by 19.3%, staying within 0.59 points of the default. Against Random Search, NSGA-II, and MOTPE, InferOpt leads on both spaces: best proxy hypervolume and lowest retention on KV, closest to the uncompressed reference at the lowest expert budget on MoE.
The method shift is worth more than the numbers
In isolation, 64% and 43% are not outrageous by compression-literature standards. The real signal is the framing the paper makes explicit: it extends the abstraction of ML auto-tuners like Ansor and Optuna from low-level tensor schedules up to high-level inference configuration. The biggest inference-optimization wins may come not from yet another algorithm, but from systematically searching the combinations of knobs we already have. For teams running their own serving stack, that means one reusable search layer that KV, MoE, and quantization can all plug into. As inference — not training — becomes where models cost the most, whoever automates the tuning first pushes marginal cost down first.