arXiv 2606.03113 introduces LEDE (Latent Ensemble for Decoding), a framework that uses offline RL to give LLM speculative decoding a "dynamic brain." The standout: LEDE-augmented speculative decoding adapts to the input, choosing the best drafter and verifier configuration on the fly, with 1.5-2× speedup over the best static configuration.
The "speculative decoding" challenge: speculative decoding uses a small "drafter" model to propose candidate tokens, and a large "verifier" model to validate them. The "drafter" and "verifier" are typically static — the same models are used for all inputs. But different inputs benefit from different configurations: simple inputs benefit from a small drafter (fast), complex inputs benefit from a large drafter (more accurate).
The LEDE fix: LEDE trains a "latent ensemble" of drafters via offline RL. The ensemble is queried at inference time, and the best drafter is selected based on the input. The selection is done by a small "router" model (1B parameters) that learns to predict which drafter is best for each input.
The benchmark: on a set of generation tasks (chat, code, math), LEDE-augmented speculative decoding hits 1.5-2× speedup over the best static configuration. The "dynamic selection" is the key — no single drafter is best for all inputs, and the dynamic selection adapts to each input.
The bigger takeaway: "dynamic configuration" is the right approach for LLM inference. The "one-size-fits-all" assumption is breaking, and the "dynamic configuration" approach is significantly more efficient. For the industry, this means "inference optimization" will move to dynamic methods, and the next round of inference improvements will come from "how smart the router is."