Agentic workloads have dragged LLM inference into an awkward zone: contexts keep growing, multi-turn interactions pile up, and the mainstream acceleration technique — speculative decoding — loses much of its power exactly there. A Shanghai Jiao Tong University team now offers a diagnosis in their DLite framework (arXiv:2610.23787): the problem is not that draft models are too weak, but that they carry a growing historical burden.

Two Problems, One Root Cause

Standard speculative decoding lets a lightweight draft model generate candidate tokens that the target model verifies in batch. The paper observes that under agentic workloads, existing draft models suffer up to an 80% degradation in acceptance length, while the draft model's own historical KV cache adds substantial system overhead.

The authors trace both problems to a single root: the ever-growing historical context. Longer contexts force the draft model to model increasingly complex dependencies, dragging prediction quality down, while the cache keeps expanding turn after turn.

The DLite Approach: Drop History, Reuse Hidden States

DLite sidesteps historical context entirely. It directly reuses contextual hidden states from the target model and performs local speculative prediction without maintaining any history on the draft side — turning an increasingly complex sequence-modeling problem into a local prediction task. A lightweight sequential head restores intra-block causal dependencies at low overhead, and a tailored two-stage training strategy aligns this heterogeneous history-free architecture.

Numbers: 2.74x on SWE-Bench, Beating DSpark

From Tables 2 and 3 of the paper (NVIDIA H800 GPUs, Qwen3-4B / Qwen3-8B targets, speculative block size K=7):

  • Agentic workloads: 2.07–2.74x end-to-end speedup; on SWE-Bench with Qwen3-8B it reaches 2.74x versus DSpark's 1.60x — about 71% higher
  • Long context (LongBench v2): 2.18–3.96x across multi-document QA, long in-context learning, code repository understanding, and long dialogue history
  • General workloads: average acceptance length of 3.64–6.48 tokens across math, coding, and chat benchmarks
  • A demo keeps the speedup on a 46,704-token long input

Notably, Liquid AI's LFM2.5-DSpark pushed edge-side speculative decoding to 2.87x with a 300M draft model back in August; two months later, the SJTU team has lifted that SOTA line again in agentic settings — the pace of iteration in this niche is striking.

Open Source, With Honest Limits

The framework is open-sourced under MIT, with a Python API, CLI, and throughput benchmark on GitHub; the companion LiteSpec training and evaluation framework also packages DFlash and DSpark baselines. Draft checkpoints for Qwen3-4B/8B/14B and Qwen3.5-4B are all released.

The README's roadmap is equally honest, though: SGLang and vLLM integrations are still on the to-do list, and support for larger target models has not landed. Eating that 2.74x in a production serving stack will have to wait for engineering follow-through — the key boundary when judging whether this is usable today.

The deeper question stands: when agents roll contexts to tens of thousands of tokens, every acceleration scheme designed for single-turn chat must re-sit its exam. DLite's answer is to throw the historical burden away — the draft model no longer keeps its own history, it hitches a ride on the target model. Anyone working on inference optimization should watch this direction.

Reference: arXiv:2610.23787 (https://arxiv.org/abs/2610.23787); code at github.com/SJTU-DAI-Sys/DLite