[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tencent-angelspec-spec-decoding-hy3-dfly":3,"news-related-926f89fc-5ed5-4170-bfac-931d3a31b6a4":41},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","7月29日,腾讯混元团队正式开源 AngelSpec 投机解码框架,覆盖从 drafter 训练到线上部署的全流程,并同步开源 Hy3-A21B 的 MTP 与 DFly drafter 权重及训练代码。框架内嵌 6 种 draft 架构,核心论文为 arXiv:2607.25852。在 Hy3-295B-A21B 上,DFly 在 4-64 并发区间相对自回归解码实现 1.98–2.40× 吞吐加速,比 DFlash 高 10.5–11.8%。","## 腾讯混元把投机解码做成了一整套开源工具链\n\n7月29日,腾讯混元团队把投机解码整套训练 + 部署栈搬到了 GitHub 上:**AngelSpec**(GitHub: Tencent\u002FAngelSpec),MIT 协议,目前 47 stars \u002F 3 forks(发布首日)。这不是单点开源一个 drafter 权重,而是把混元自研的两类 drafter、统一的训练 pipeline、vLLM \u002F SGLang 部署后端,以及配套的 paper \u002F docs 一起打包放出来,顺带把 Hy3-295B-A21B 上跑出来的核心 benchmark 也贴在了 README 里。\n\n### 它到底在解决什么问题\n\nLLM 自回归解码是「一次前向只能出一个 token」的瓶颈。投机解码(speculative decoding)用一个轻量 drafter 先猜一串候选 token,再让目标模型用一次前向整体验证、提交可接受的前缀和奖励 token,从而把「一次迭代推进多个 token」变成现实。AngelSpec 把这件事拆成三层:\n\n- **训练层**:为不同 workload 定制互补的 drafter——MTP drafter 用丰富、对话风格的数据训练,适合高熵开放式对话;块扩散 drafter(DFly)用代码 + 数学数据加强,吃结构化长可预测片段的红利。\n- **架构层**:提出 **DFly**——一个块扩散框架,hybrid target-conditioning backbone + 前驱条件自回归头,既保留 DFlash 的并行吞吐,又把「位置间依赖」这件事修了回来。\n- **推理层**:集成 **D-cut**,把 target 验证算力视作 batch 级共享资源,根据在线负载与各请求的置信度动态分配验证深度,避免固定验证深度在高并发下浪费 batch 容量。\n\n### 关键数字(Hy3-295B-A21B, TP=8)\n\n- **DFly 平均接受长度 4.79**(MTP 3.00 \u002F DFlash 3.69),相当于相对 MTP +59.7%、相对 DFlash +29.8%。\n- **吞吐加速**:在 4-64 并发下,**DFly 相对自回归解码实现 1.98–2.40× 加速,比 DFlash 再高 10.5–11.8%**。\n- **结构化场景优势最大**:HumanEval 上平均接受长度 5.52,Math500 上 5.23,GSM8K 上 5.53。\n- **线上流量**:在 Hy3 live traffic(TP=8, 8× H20, 并发 2-64)上,DFly 在高置信前缀上跑赢 D-cut 的 aggregate throughput 拐点(并发 ≥48 后饱和)。\n\nQwen3-8B 同样被验证,DFly 平均接受长度 5.41,所有 5 个数学\u002F代码 benchmark 上都是第一;MT-Bench 上 DSpark(5.32 平均)略好,也契合作者对「DFly 主打代码\u002F数学,对话交给 MTP」的定位。\n\n### 这件事值得多看一眼的几个角度\n\n**1. 框架一体化的方向感。** AngelSpec 不是「再加一种 drafter」,而是把 6 种 draft 架构(DFly \u002F DFlash \u002F DFlare \u002F Eagle3 \u002F DSpark \u002F MTP)统一塞进同一套训练 pipeline——切架构是改配置,不再重写训练代码。配合 Mooncake 做 disaggregated hidden-state 生成,推理引擎和工作进程是两组 GPU,通过 tensor store 解耦,能各自独立扩缩容。这条路和 LightSeek Foundation 的 TorchSpec 一脉相承,但工程化完整度明显上了一个台阶。\n\n**2. 「workload heterogeneity」是这次的核心论点。** 论文开篇就讲:不要在均匀数据混合上训一个「通用 drafter」——对话高熵场景需要 MTP 这种短 draft,代码\u002F数学低熵长可预测场景需要块扩散。AngelSpec 用同一框架训出两种互补的 drafter,再用 D-cut 在推理时按请求\u002F在线负载动态分配验证深度。这是一条「训练数据 + 架构 + 推理策略」协同设计的路线,而不是单点优化。\n\n**3. DFly 的两个具体技术贡献值得拎出来。** 一是 hybrid target-conditioning:DFlash 的全局非线性变换(ct)+ DFlare 的 per-layer 标量加权(ft(i))做加性组合 + RMSNorm,既保留跨层交互的表征能力,又给每一层 draft 单独的 target view,代价只是 D×T 个标量权重(softmax 后可预计算)。二是 hidden-correction head:在并行 backbone 之后挂一个轻量顺序头,把位置 i 的 draft 表示和「位置 i-1 采样出的 token embedding」一起喂进 SwiGLU,得到位置 i 的条件分布,把并行预测从「边际分布」升级成「前缀条件分布」。消融实验里 hidden-correction 比 Markov head 在所有 benchmark 上都更优。\n\n**4. 训练侧的两个易被忽略的工程化能力。** 一是 TTT(Training-Time Test):MTP 块在训练时按自回归方式展开 D 步,每个深度都用 argmax 预测喂下一步,显式制造 train-inference mismatch 后再学着恢复——这是 EAGLE-3 的做法,在 MoE + 长上下文场景下尤其值钱。二是 online acceptance evaluation:训练过程中挂一个真投机解码评估服务,直接读当前 checkpoint 在 serving engine 里的接受长度和按位接受率,而不是离线算 proxy 指标。\n\n**5. 实际的硬件和数据成本。** 8 GPU 单机 quickstart(4 推理 + 4 训练)就能跑 Qwen3-8B DFly。多节点示例直指 Hy3(DFly 训出 ,MTP 训出 ),并且同时给了 High-think 变体()以覆盖推理场景。文档里明确提示 CUDA 12 主机要装对应版本的 wheel(PyPI 默认 CUDA 13),这是个真实会踩的坑。\n\n### 所以呢\n\nLLM 推理优化的竞争点已经从「训出更大的模型」明显转向「同样模型如何跑得更便宜」。AngelSpec 给出的不是某个新数字,而是一套「workload-aware 训练 + 异构 drafter + 自适应验证深度」的完整打法,把投机解码从 paper trick 推进到 production stack。腾讯混元这次走的是 EAGLE \u002F DFlash 之后开源生态里少见的「全家桶式」路线——对做 LLM serving 的工程团队而言,这意味着少走很多重复造轮子的路;对研究者而言,MoE + 长上下文 + 在线 batch 这三个真实生产约束,终于有了可复现的基线。\n\n如果只看一个数字:**DFly 在 Hy3-A21B 上把平均接受长度从 3.00(MTP)拉到 4.79,对应 1.98–2.40× 自回归加速**——这是当下一线大模型(295B MoE)上,公开报告中能拿到的最好投机解码结果之一。开源意味着同行可以直接在自己的 serving 集群上复现这条数字。","https:\u002F\u002Fgithub.com\u002FTencent\u002FAngelSpec","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":25,"name":26,"slug":26,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"76c7fd64-447d-4de0-9cf1-4e80db4661ca","en","AngelSpec: DFly speeds Hy3-A21B by 1.98-2.40x, open-sourced","On July 29, Tencent Hunyuan released AngelSpec, an open-source, torch-native framework unifying MTP and block-parallel speculative decoding training through online deployment. It bundles 6 draft architectures, vLLM and SGLang backends, and releases Hy3-A21B MTP and DFly drafter weights. On Hy3-295B-A21B (TP=8), DFly achieves 1.98–2.40× throughput over autoregressive decoding across concurrency 4–64, and 10.5–11.8% higher than DFlash.","## Tencent Hunyuan Turns Speculative Decoding Into a Full Open-Source Stack\n\nOn July 29, the Tencent Hunyuan team put the entire speculative decoding train-and-deploy stack on GitHub: **AngelSpec** (Tencent\u002FAngelSpec), MIT-licensed, 47 stars \u002F 3 forks on day one. This is not a single-point release of a drafter weight — it bundles two proprietary drafters, a unified training pipeline, vLLM and SGLang serving backends, the companion paper, and the headline Hy3-295B-A21B benchmark numbers directly in the README.\n\n### What problem does it solve\n\nAutoregressive LLM decoding is bottlenecked by the fact that one forward pass produces one token. Speculative decoding uses a lightweight drafter to propose a candidate sequence, then has the target model verify the entire block in a single forward pass via rejection sampling, committing the accepted prefix plus a bonus token. Each iteration then advances multiple tokens. AngelSpec decomposes the problem into three layers:\n\n- **Training layer**: tailor complementary drafters for different workloads — an MTP drafter trained on rich conversational data for high-entropy open-ended chat; a block-diffusion drafter (DFly) strengthened with code + math data to harvest longer predictable spans in structured generation.\n- **Architecture layer**: introduce **DFly**, a block-diffusion framework that combines a hybrid target-conditioning backbone with a predecessor-conditioned autoregressive head — keeping DFlash's parallel throughput while restoring intra-block dependency modeling.\n- **Inference layer**: integrate **D-cut**, which treats target-verification compute as a shared batch-level resource and dynamically reallocates verification depth according to online load and per-request prefix confidence, avoiding the waste of fixed verification depth under high concurrency.\n\n### Key numbers (Hy3-295B-A21B, TP=8)\n\n- **DFly mean accepted length 4.79** (MTP 3.00, DFlash 3.69), a +59.7% gain over MTP and +29.8% over DFlash.\n- **Throughput speedup**: at concurrency 4–64, **DFly delivers 1.98–2.40× speedup over autoregressive decoding, plus 10.5–11.8% over DFlash**.\n- **Largest gains in structured domains**: HumanEval mean accepted length 5.52, Math500 5.23, GSM8K 5.53.\n- **Live traffic**: on Hy3 production traffic (TP=8, 8× H20, concurrency 2–64), DFly surpasses D-cut's aggregate-throughput inflection point — DFly saturates beyond concurrency 48 while D-cut continues to convert additional load.\n\nQwen3-8B was validated the same way: DFly averages 5.41 mean accepted length, first on all 5 math\u002Fcode benchmarks; DSpark (5.32 average) edges ahead on MT-Bench, consistent with the authors' positioning that DFly targets code\u002Fmath while chat falls to MTP.\n\n### Five angles worth a closer look\n\n**1. The direction is framework unification.** AngelSpec is not \"yet another drafter.\" It unifies 6 draft architectures (DFly, DFlash, DFlare, Eagle3, DSpark, MTP) under one training pipeline — switching architectures is a config change, not a rewrite. Disaggregated hidden-state generation runs inference engines and workers as separate GPU groups communicating through a Mooncake tensor store, so each can scale independently. The lineage is LightSeek Foundation's TorchSpec, but engineering completeness is visibly higher.\n\n**2. \"Workload heterogeneity\" is the paper's central thesis.** The paper opens by arguing against training one universal drafter on a uniform mixture: high-entropy chat wants short MTP drafts, low-entropy code\u002Fmath want long block-parallel drafts. AngelSpec trains two complementary drafters in the same framework, then uses D-cut at inference time to allocate verification depth per request, per online load. It is a co-design across training data, architecture, and inference policy — not a single-point optimization.\n\n**3. Two concrete technical contributions of DFly are worth singling out.** First, hybrid target-conditioning: DFlash's global nonlinear transformation (ct) plus DFlare's per-layer scalar weighting (ft(i)) are combined additively with RMSNorm — preserving cross-layer representational capacity while giving each draft layer its own target view, at a cost of only D×T scalar weights (softmax coefficients can be precomputed after training). Second, the hidden-correction head: a lightweight sequential head mounted after the parallel backbone that takes the position-i draft representation together with the token embedding sampled at position i-1 through a SwiGLU, turning the position-wise marginal prediction into a prefix-conditioned distribution. Ablations show hidden-correction outperforms the Markov head on every benchmark.\n\n**4. Two easily overlooked engineering capabilities on the training side.** First, TTT (Training-Time Test): the MTP block is autoregressively unrolled for D steps during training, each depth fed by the argmax prediction of the previous depth — explicitly manufacturing the train-inference mismatch and then learning to recover. This is the EAGLE-3 approach, especially valuable under MoE + long-context regimes. Second, online acceptance evaluation: a real speculative-decoding evaluation server runs during training, reading mean accepted length and per-position acceptance directly from the serving engine on the current checkpoint — not offline proxy metrics.\n\n**5. The actual hardware and data cost.** 8-GPU single-node quickstart (4 inference + 4 training) is enough to run Qwen3-8B DFly. The multi-node examples target Hy3 directly (DFly produces AngelSlim\u002FHy3-DFly-Block8, MTP produces AngelSlim\u002FHy3-MTP-TTT3), plus a High-think variant (Hy3-DFly-Block8-Think-High) to cover reasoning-mode serving. The docs explicitly warn that CUDA 12 hosts need CUDA-matched wheels (PyPI defaults to CUDA 13) — a real footgun.\n\n### So what\n\nThe optimization battlefield in LLM inference has visibly shifted from \"train a bigger model\" to \"how do we run the same model cheaper.\" AngelSpec delivers not a single number but a complete playbook of workload-aware training + heterogeneous drafters + adaptive verification depth — pushing speculative decoding from paper trick to production stack. Tencent Hunyuan's choice to go full-bundle — unusual in the post-EAGLE \u002F post-DFlash open-source ecosystem — means LLM serving teams skip a lot of wheel-rebuilding. For researchers, the combination of MoE + long-context + online-batch finally has a reproducible baseline.\n\nIf you only take one number: **DFly lifts mean accepted length on Hy3-A21B from 3.00 (MTP) to 4.79, equivalent to 1.98–2.40× autoregressive speedup** — among the best publicly-reported speculative decoding results on a frontier-class MoE today. Open-sourcing means peers can reproduce this number on their own serving clusters.","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00Z","2026-07-29T16:05:16.368300Z","2026-07-29T16:05:16.368317Z",true,"agent",147,{"items":42},[43,48,53,58,63,68],{"id":44,"title":45,"news_slug":46,"published_at":47},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"92433e6b-113a-4ada-af77-fbb8995a9850","LFM2.5-DSpark 开源:300M 草稿模型让端侧推理快 2.87 倍,输出零损耗","lfm2-5-dspark-draft-models","2026-08-21T21:10:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":69,"title":70,"news_slug":71,"published_at":72},"894787b7-8799-4c22-8e20-5e04db8f7181","DeepSeek V4 GA 把分时电价塞进大模型 API:CSA + HCA 撑起 1M 上下文,峰值翻倍只是商业答案","deepseek-v4-ga-time-of-use-pricing","2026-07-24T02:00:00+00:00"]