[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-specpilot-agentic-speculative-decoding":3,"topics-all":38,"news-related-64088ca6-c244-474c-93ce-cdbfe55de6d6":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"64088ca6-c244-474c-93ce-cdbfe55de6d6","投机解码没用满?agent循环把加速比挖到10.87倍","arXiv 新论文 SpecPilot 用 agent 循环自动调优投机解码:成本模型找差距、并行探索、逐 token 防作弊。实测 4 模型、3 个 draft、2 种 GPU、5 基准下,单请求加速比 4.46 到 10.87 倍,batch 1.94 到 3.93,全程无损,代码开源。","投机解码(speculative decoding)是当前 LLM 推理最实用的加速手段之一:让轻量的 draft 模型先草拟一段 token,目标大模型一次前向验证完,省掉自回归逐 token 的等待;并行 draft 模型进一步把提速做大,一次前向就草拟一整块。但 arXiv 新论文 SpecPilot 指出,各家发布出来的投机解码系统,实际跑出的加速往往只是「应得加速」的一部分——剩下的余量,被三个工程问题吃掉了。\n\n## 三个被浪费的加速空间\n\n论文把浪费归纳为三点。第一,单请求场景下,每一步验证只接受少量 token,却要读完目标模型的全部权重,读权重的开销远超接受的收益。第二,一旦上 batch,每个请求都要做一遍验证,加速比相对自回归解码大致减半。第三,投机解码强部署相关:换一个目标模型、draft 模型、GPU 或 serving 配置,最优参数组合就变了,不存在万能配置——这正是一般论文给不了、工程团队又没人力逐一去调的部分。\n\n## 让 agent 循环自己找优化\n\n作者的解法是把「调优」这件事本身交给一个 agent 循环。每一轮,先用成本模型定位当前系统与性能上限之间最大的差距;多个并行 agent 各自探索一个优化方向,把成功的集成进系统;另有专门的 kernel 子 agent,负责调优离内存带宽上限最远的内核。两道安全设计值得注意:一是 token forcing,把每个输出 token 与原始系统逐一对照,专门用来抓 agent 的 reward hacking;二是结构化记忆,把验证过的机制带到新一轮循环和新部署,不重复交学费。这里的原系统(stock)即未加 SpecPilot 的 DFlash2 或 DSpark 原版,与优化后引擎跑同一 serving。\n\n## 实测:同样的硬件,加速比翻倍以上\n\n这个循环挖出了一批通用机制:无损的 12-bit 权重格式、面向混合循环架构目标的 tree verification、两级验证,以及三种 batch 机制。评估覆盖 4 个目标模型(公开表格含 Qwen3.8-27B 与 Muse-Glimmer-30B)、3 个 draft 模型(DFlash2 与 DSpark 等)、2 种 GPU 平台(H100 与 RTX PRO 6000)和 5 个基准(gsm8k、humaneval、math500、mbpp、mt-bench),口径为每请求生成 2048 个新 token、按请求取加速比几何平均,batch 并发 16、20、24:单请求下,相对自回归解码的加速比从原系统的 4.46 倍提到 10.87 倍;batch 下从 1.94 倍提到 3.93 倍,每个部署都翻倍以上,单请求每步提交的 token 数也从约 4.4-5.2 提到约 12.6-13.1。全程无训练、无质量损失:解码保持贪心,每个 token 由目标模型拍板,严格 lossless。\n\n代码已按 Apache 2.0 开源(基于 SGLang 衍生),仓库放出 6 个部署的引擎与复现脚本,论文由 Kunjun Li 等 10 位作者完成:[arXiv 论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.02501)、[GitHub 仓库](https:\u002F\u002Fgithub.com\u002FStargazerX0\u002FSpecPilot)。\n\n## 所以呢\n\n这篇论文值得记的不是 10.87 这个数字,而是它验证的工程范式:部署特定的系统优化,人力成本高到没人做得全,而 agent 循环可以把这块「余量」自动化挖出来。更有意思的是作者对自家 agent 的态度——产出再好,也要逐 token 对照原系统防作弊。对推理成本敏感的团队来说,与其等下一代模型,不如先算算现有部署里还剩几倍没挖出来。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.02501","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"5239b31a-14f2-421e-b03e-00705efba2a7","en","SpecPilot: Agent Loop Tunes Speculative Decoding to 10.87x","Agentic loop finds lossless speculative decoding speedups per deployment: 4.46x to 10.87x single-request, 1.94x to 3.93x batched, zero quality loss.","Speculative decoding is one of the most practical LLM inference speedups today: a lightweight drafter proposes a block of tokens and the target model verifies them in one forward pass, cutting the serial waiting of autoregressive decoding; parallel drafters push the speedup further by drafting a whole block in one forward pass. A new arXiv paper, SpecPilot, argues that released speculative decoding systems deliver only part of the speedup they could — the remaining headroom is eaten by three engineering problems.\n\n## Three sources of wasted speedup\n\nThe paper names them. First, with a single request, each verification step accepts only a few tokens yet reads every target weight, so the weight-reading cost far exceeds what the accepted tokens earn. Second, once batching kicks in, per-request verification roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific: change the target model, drafter, GPU or serving regime and the optimal parameter mix changes — no universal tuning exists, which is exactly the part generic papers cannot deliver and engineering teams lack the manpower to do exhaustively.\n\n## An agent loop that tunes itself\n\nThe authors hand the tuning itself to an autonomous agentic loop. Each round, a cost model locates the largest gaps between the current system and its ceiling; parallel agents each explore one optimization direction and integrate the successful ones; a dedicated kernel sub-agent tunes the kernels furthest from their memory-bandwidth limit. Two safety designs stand out. Token forcing checks every output token against the stock system specifically to catch reward hacking, and a structured memory carries validated mechanisms to new rounds and new deployments instead of relearning them. Here stock means the released drafter system (DFlash2 or DSpark) without SpecPilot, served on the same engine.\n\n## Measured: more than double on the same hardware\n\nThe loop surfaced a set of general mechanisms: a lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification, and three batching mechanisms. The evaluation spans four target models (the public table includes Qwen3.8-27B and Muse-Glimmer-30B), three drafters (DFlash2 and DSpark among them), two GPU platforms (H100 and RTX PRO 6000) and five benchmarks (gsm8k, humaneval, math500, mbpp, mt-bench), with each request generating 2048 new tokens, per-request speedups geometrically averaged, and batch runs at concurrency 16, 20 and 24. Without training or quality loss — decoding stays greedy and the target model decides every token, strictly lossless — the speedup over AR rises from stock's 4.46x to 10.87x with one request, and from 1.94x to 3.93x under batching, more than doubling in every deployment. Tokens committed per verify step in single-request settings climb from roughly 4.4-5.2 to 12.6-13.1.\n\nThe code is open-sourced under Apache 2.0 (derived from SGLang), shipping the engine found for each of six deployments plus scripts to serve, benchmark and verify them. The paper lists ten authors, led by Kunjun Li: [arXiv paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.02501), [GitHub repo](https:\u002F\u002Fgithub.com\u002FStargazerX0\u002FSpecPilot).\n\n## So what\n\nThe takeaway worth keeping is not the 10.87x number but the engineering pattern it validates: deployment-specific systems optimization is too costly for humans to complete, and an agent loop can mine that headroom automatically. Just as telling is how the authors treat their own agent — however good the output, every token is checked against the stock system to catch cheating. For teams sensitive to inference cost, before waiting for the next model generation, it may be cheaper to measure how many multiples are still buried in the current deployment.","specpilot-agentic-speculative-decoding","2026-11-04T13:15:17Z","2026-11-04T13:15:23.364416Z","2026-11-04T13:15:23.364429Z",true,"agent",47,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"6784d3bd-26c1-4fd5-a2e5-8c7b9e591dae","SmoothAgent 把上下文变换「提前做」：Agent 长链路 TTFT 砍到原来的 1\u002F12","smoothagent-ttft-12x","2026-07-23T03:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"83ef24e7-e2f5-4aec-8b4e-88bd10f8c28c","SelfCompact 把\"何时压缩\"从 LLM 智能体身上卸下:6 基准 18.1 分提点,token 成本砍 30-70%","selfcompact-jhu-when-to-compress-rubric","2026-06-23T08:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"5988b398-8533-4af5-9aca-ff6e51e9f429","Parallel-Synthesis：让 KV Cache 成为 Agent 多分支的「原生合成接口」，TTFT 提速 2.5–11×","parallel-synthesis-kv-cache-agent-2-5-11x","2026-06-15T14:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"c0fdcd77-4585-4b04-b042-ddf1789c2158","AI模型发布逻辑正在重写：2026年5月，四条主线改变竞争规则","may-2026-4-main-lines-competition-rewrite","2026-05-27T00:15:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a550dbc5-240b-4747-b625-9aa874aad444","工具拉满的AI停在0.48倍:会提问的做到1.13倍","kernelseek-question-driven-cuda-kernel-agent","2026-11-03T09:05:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"125a6c5c-2930-40a5-9058-bc873b2521df","塔吉克语工具模型:CPT清零工具调用,SFT拉回64分","saidzoda-tajik-lfm-2-6b-tool-calling","2026-11-02T19:11:09+00:00"]