Tencent Hunyuan Turns Speculative Decoding Into a Full Open-Source Stack

On July 29, the Tencent Hunyuan team put the entire speculative decoding train-and-deploy stack on GitHub: AngelSpec (Tencent/AngelSpec), MIT-licensed, 47 stars / 3 forks on day one. This is not a single-point release of a drafter weight — it bundles two proprietary drafters, a unified training pipeline, vLLM and SGLang serving backends, the companion paper, and the headline Hy3-295B-A21B benchmark numbers directly in the README.

What problem does it solve

Autoregressive LLM decoding is bottlenecked by the fact that one forward pass produces one token. Speculative decoding uses a lightweight drafter to propose a candidate sequence, then has the target model verify the entire block in a single forward pass via rejection sampling, committing the accepted prefix plus a bonus token. Each iteration then advances multiple tokens. AngelSpec decomposes the problem into three layers:

  • Training layer: tailor complementary drafters for different workloads — an MTP drafter trained on rich conversational data for high-entropy open-ended chat; a block-diffusion drafter (DFly) strengthened with code + math data to harvest longer predictable spans in structured generation.
  • Architecture layer: introduce DFly, a block-diffusion framework that combines a hybrid target-conditioning backbone with a predecessor-conditioned autoregressive head — keeping DFlash's parallel throughput while restoring intra-block dependency modeling.
  • Inference layer: integrate D-cut, which treats target-verification compute as a shared batch-level resource and dynamically reallocates verification depth according to online load and per-request prefix confidence, avoiding the waste of fixed verification depth under high concurrency.

Key numbers (Hy3-295B-A21B, TP=8)

  • DFly mean accepted length 4.79 (MTP 3.00, DFlash 3.69), a +59.7% gain over MTP and +29.8% over DFlash.
  • Throughput speedup: at concurrency 4–64, DFly delivers 1.98–2.40× speedup over autoregressive decoding, plus 10.5–11.8% over DFlash.
  • Largest gains in structured domains: HumanEval mean accepted length 5.52, Math500 5.23, GSM8K 5.53.
  • Live traffic: on Hy3 production traffic (TP=8, 8× H20, concurrency 2–64), DFly surpasses D-cut's aggregate-throughput inflection point — DFly saturates beyond concurrency 48 while D-cut continues to convert additional load.

Qwen3-8B was validated the same way: DFly averages 5.41 mean accepted length, first on all 5 math/code benchmarks; DSpark (5.32 average) edges ahead on MT-Bench, consistent with the authors' positioning that DFly targets code/math while chat falls to MTP.

Five angles worth a closer look

1. The direction is framework unification. AngelSpec is not "yet another drafter." It unifies 6 draft architectures (DFly, DFlash, DFlare, Eagle3, DSpark, MTP) under one training pipeline — switching architectures is a config change, not a rewrite. Disaggregated hidden-state generation runs inference engines and workers as separate GPU groups communicating through a Mooncake tensor store, so each can scale independently. The lineage is LightSeek Foundation's TorchSpec, but engineering completeness is visibly higher.

2. "Workload heterogeneity" is the paper's central thesis. The paper opens by arguing against training one universal drafter on a uniform mixture: high-entropy chat wants short MTP drafts, low-entropy code/math want long block-parallel drafts. AngelSpec trains two complementary drafters in the same framework, then uses D-cut at inference time to allocate verification depth per request, per online load. It is a co-design across training data, architecture, and inference policy — not a single-point optimization.

3. Two concrete technical contributions of DFly are worth singling out. First, hybrid target-conditioning: DFlash's global nonlinear transformation (ct) plus DFlare's per-layer scalar weighting (ft(i)) are combined additively with RMSNorm — preserving cross-layer representational capacity while giving each draft layer its own target view, at a cost of only D×T scalar weights (softmax coefficients can be precomputed after training). Second, the hidden-correction head: a lightweight sequential head mounted after the parallel backbone that takes the position-i draft representation together with the token embedding sampled at position i-1 through a SwiGLU, turning the position-wise marginal prediction into a prefix-conditioned distribution. Ablations show hidden-correction outperforms the Markov head on every benchmark.

4. Two easily overlooked engineering capabilities on the training side. First, TTT (Training-Time Test): the MTP block is autoregressively unrolled for D steps during training, each depth fed by the argmax prediction of the previous depth — explicitly manufacturing the train-inference mismatch and then learning to recover. This is the EAGLE-3 approach, especially valuable under MoE + long-context regimes. Second, online acceptance evaluation: a real speculative-decoding evaluation server runs during training, reading mean accepted length and per-position acceptance directly from the serving engine on the current checkpoint — not offline proxy metrics.

5. The actual hardware and data cost. 8-GPU single-node quickstart (4 inference + 4 training) is enough to run Qwen3-8B DFly. The multi-node examples target Hy3 directly (DFly produces AngelSlim/Hy3-DFly-Block8, MTP produces AngelSlim/Hy3-MTP-TTT3), plus a High-think variant (Hy3-DFly-Block8-Think-High) to cover reasoning-mode serving. The docs explicitly warn that CUDA 12 hosts need CUDA-matched wheels (PyPI defaults to CUDA 13) — a real footgun.

So what

The optimization battlefield in LLM inference has visibly shifted from "train a bigger model" to "how do we run the same model cheaper." AngelSpec delivers not a single number but a complete playbook of workload-aware training + heterogeneous drafters + adaptive verification depth — pushing speculative decoding from paper trick to production stack. Tencent Hunyuan's choice to go full-bundle — unusual in the post-EAGLE / post-DFlash open-source ecosystem — means LLM serving teams skip a lot of wheel-rebuilding. For researchers, the combination of MoE + long-context + online-batch finally has a reproducible baseline.

If you only take one number: DFly lifts mean accepted length on Hy3-A21B from 3.00 (MTP) to 4.79, equivalent to 1.98–2.40× autoregressive speedup — among the best publicly-reported speculative decoding results on a frontier-class MoE today. Open-sourcing means peers can reproduce this number on their own serving clusters.