[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-speculative-decoding-bpe-tokenizer-ceiling":3,"topics-all":38,"news-related-8d830f21-a44f-45b8-be6c-eed523d060dc":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"8d830f21-a44f-45b8-be6c-eed523d060dc","投机解码的天花板是分词器:3.4M参数换2.4倍","一篇 arXiv 论文指出,主流投机解码都在目标模型自己的词表里草稿,候选树一个节点就是一个 token,单位被钉死在 1。作者另建一张独立短语词表,用 3.434M 参数的 MLP 头预测一到三词短语,目标模型全程冻结,MATH-500 上跑出 2.444 倍。论文自曝:加速多来自标点与 LaTeX 骨架。","投机解码这两年一直在卷更准的草稿模型、更深的树、更宽的树,却很少有人去动真正卡住所有方法的量:一次草稿决策能换回多少个目标 token。这篇论文把它叫做「单位」,并指出它被 BPE 的预分词规则钉死在 1 上——Qwen3-4B 词表里「United States of America」是四个 token,任何相邻两个都不构成一条词表项,多词短语对「在目标词表里草稿」的方法都不可达。作者 Madhusudhana Naidu,11 月 9 日提交到 [arXiv](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12464)。\n\n## 一个被默认的常数\n\n诊断从一个公式开始:每个前向能「存下」的 token 数 = 1 + landing × (每次被接受的草稿单位换回的目标 token 数)。前面那个 1 是目标模型自己送的白菜,后面被乘的才是单位。Medusa、EAGLE-3、DFlash、DSpark 全都在目标模型自己的词表里草稿,候选树的一个节点就是一个目标 token,单位恒为 1,能动的只剩节点数量。\n\n作者团队自己先撞上了这堵墙:在 Qwen3-4B 上做的词级草稿器,landing 做到 0.711,看着不算差,但每次被接受的猜测只换回 1.049 个目标 token,基本贴着 1。论文的总结很硬:不是接受率不行,是单位太短。\n\n单位为什么短?BPE 在学合并规则之前先按空白切分,再多合并训练也造不出跨词的 token。SuperBPE 说得很直白,空白只是历史上为切形态学子词而设的规矩;它走改目标分词器的路,代价是从零预训练 8B 模型、约 330B token,而这篇论文不碰目标模型。EAGLE-3 的天花板也由单位决定:草稿头用截断词表,151,936 个 id 里只留 32,000 个(21.06%),外面的目标 token 任何树深树宽都写不出来。\n\n## 换单位:另建一张短语词表\n\n做法是:目标模型和分词器全程冻结,两层 LayerNorm 的 MLP 头读目标隐状态,预测一个一到三词的短语,短语是独立草稿词表里的一条目;命中后用贪心最长匹配展开回目标 token,多个候选在一次前向里靠前缀树掩码一起验证。因为只接受与目标自身 argmax 一致的候选,输出与贪心解码一致。\n\n参数账很干净:头的固定开销是 1,377,792 个参数(d_model=2560、H_DIM=256),再加每个词表条目 257 个输出行参数。于是词表大小就是参数预算本身:8,000 条对应 3,433,792 个参数、占 Qwen3-4B 的 0.0854%,输出层占 59.9%;80,000 条对应 21,937,792 个、0.5454%,输出层占 93.7%。十分之一的词表只买到六分之一的参数。\n\n21.94M 的头(占 Qwen3-4B 的 0.545%)在 GSM8K 上 2.391 倍、MATH-500 上 2.444 倍、MT-Bench 上 1.544 倍(它训练时只见过数学蒸馏,MT-Bench 是零样本迁移)。缩到 8,000 条的头是 3.434M 参数、0.0854%,比 EAGLE-3 的 218.2M 草稿器小 64 倍,却保住大个头 GSM8K 上 92.3% 的每趟 token 数。\n\n放到同一个分母上比更有意思:22M 的头在 GSM8K 和 MATH-500 上分别相当于 EAGLE-3 tree-16 的 102.3% 和 108.5%,但每个数据集上都没追上 tree-60(91.5%、98.1%,MT-Bench 上只剩 66.6%)。论文对此不遮掩:它讲的是同等工作点上的参数效率,不是新的速度纪录。\n\n## 加速到底从哪来\n\n这是论文最有价值的自我拆解,也是最反直觉的一处。\n\n作者统计了草稿词表里装着什么。8,000 条头里 3,997 条是多词条目,只有 467 条(11.7%)是纯字母加空白;43.9% 含 LaTeX 字符,55.0% 含数字,24.7% 含换行。频率最高的条目在两个头之间完全一致,都是 Markdown 和 LaTeX 脚手架。\n\n再统计真正入账的 token 来自哪类条目:标点占 31.4%(GSM8K)、41.0%(MT-Bench),LaTeX 占 26.0%、11.7%,纯英文词只有 7.3% 和 22.6%;数学题上标点加 LaTeX 合计 57%–63%,59% 的入账来自三词条目——按上面那份普查,那些多是 LaTeX 碎片。论文自己的话是:这个方法在收割数学散文的低熵排版骨架。\n\n跨域时这张账单直接翻面。换到 MT-Bench,LaTeX 从 41.3% 掉到 11.7%,纯词从 5.5% 涨到 22.6%,单词条目从 12.4% 涨到 35.7%,landing 从 0.817 掉到 0.509。论文的解释是:对话里没有排版骨架可收割,草稿只好退回猜常见短词。\n\n还有个更扎人的细节:8,000 条的头在 MT-Bench 上每个 N 都打赢了 40,000 条的头(最好 1.653 对 1.588)。多出来的 32,000 条不只是没用——论文说它们「有害」,每一个都是在同一份监督上训练的竞争 logit。\n\n## 另一头:词被切碎的语言\n\n论文认为真正的战场不在英语,而在词被切得最碎的语言。Qwen3-4B 的分词器在 23 种语言上量下来:英语 1.38 token\u002F词,印地语 5.00,泰卢固语 11.73,印度语系平均 8.68;而 151,669 个 token 里印度系文字只有 253 个(0.17%),以词为单位的草稿一次决策就能白赚好几个 token。但论文自己承认这一档基本没跑:印地语系的实验用的是 Gemma-3-1B,而 Gemma-3 每泰卢固语词只花 3.05 个 token,Qwen3 要 11.73,差 3.8 倍——正好落在空间最小的那一侧。\n\n## 论文自己承认的账\n\n局限章节比多数同类工作诚实,每条都是实测出来的。分母很脆:仅仅换一个 transformers 库版本就值 5.7%–9.1% 的加速数字,同一张卡三次运行在 MATH-500 上有 5.7% 的抖动。一张卡、一个种子、50\u002F50\u002F40 道题,没有置信区间。在线自适应「还不是一个结果」:landing 单调从 0.736 升到 0.759 再到 0.764,可加速没跟上(2.180 → 2.214 → 2.201),而且没有冻结对照组,那条曲线分不清是自适应还是流在漂。\n\n## 所以呢\n\n真正的贡献可能不是 2.444 倍,而是把「单位」从常数变成了变量:只要接受率不是瓶颈,天花板就取决于一次草稿决策能换回几个目标 token,而这个数被分词器的空白预分词规则卡着。这也提醒我们多问一句——它换回来的是语义,还是格式?\n\n工程上这条路适用面很具体:数学、代码这类格式规整的流量上,一个 3.4M 参数、占 4B 目标 0.0854% 的小头就能吃到两倍多吞吐;对话流量上,论文的账本说明收益会掉到 1.5 倍左右。想动手,先量的不是树宽,而是自己分词器的 fertility。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12464","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"045c011e-e2bb-45ce-bdd6-0c927f8a3b87","token-efficiency",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"deb8456f-9849-4d19-be78-dc0769304651","en","The Ceiling of Speculative Decoding Is the Tokenizer","arXiv 2611.12464: mainstream speculative decoders pin the draft unit at one token. A 3.434M-parameter phrase head hits 2.444x on MATH-500.","For two years the speculative decoding literature has been chasing better draft models, deeper trees and wider trees. Almost nobody has touched the quantity that actually caps every method: how many target tokens one draft decision can bank. This paper calls that quantity the \"unit\", and shows it is nailed to 1 by a structural property of BPE. In Qwen3-4B's vocabulary \"United States of America\" is four tokens and no adjacent pair of them exists as one entry, so a multi-word phrase is out of reach of any method that drafts in the target's own vocabulary. The paper is authored by Madhusudhana Naidu and was [submitted to arXiv](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12464) on November 9.\n\n## A constant nobody questioned\n\nThe diagnosis starts from a formula: the tokens a forward pass can bank equal 1 + landing x (target tokens banked per accepted draft unit). That leading 1 is the target model's own bonus token; what gets multiplied is the unit. Every mainstream method — Medusa, EAGLE-3, DFlash and DSpark — drafts in the target's own vocabulary, so one node of the candidate tree is one target token. The unit is pinned at 1 and the only lever left is the number of nodes.\n\nThe authors hit this wall themselves first. Their earlier word-level drafter on Qwen3-4B reached a landing of 0.711, which does not look bad, but still banked only 1.049 target tokens per accepted guess — right against the floor of 1. The paper's summary is blunt: acceptance was not the limit; the unit was.\n\nWhy is the unit short? BPE's pretokenizer splits on whitespace before merges are learned, so no amount of BPE training can produce a multi-word token. SuperBPE makes the same argument from the tokenizer side: whitespace is not a reliable delimiter of meaning, and the restriction is an inherited convention rather than a design requirement. It solves the same problem by changing the target's own tokenizer, at the cost of pretraining from scratch — 8B models and roughly 330B tokens — whereas this work never touches the target. EAGLE-3's ceiling is set by the same unit: its draft head works over a truncated vocabulary of 32,000 ids out of 151,936 on Qwen3-4B, or 21.06%, and a target token outside that set is undraftable at any tree size or depth.\n\n## Changing the unit: a separate phrase vocabulary\n\nThe mechanism: the target model and its tokenizer stay frozen, while a two-LayerNorm MLP head reads the target's hidden state and predicts a one- to three-word phrase as a single entry in a separate draft vocabulary. A greedy longest-match expansion maps that entry back into the target's own tokens, and multiple candidates are verified in one forward pass behind a prefix-tree mask. Because a candidate is accepted only while it matches the target's own argmax, the served output is identical to greedy decoding.\n\nThe parameter arithmetic is clean: the head's fixed cost is 1,377,792 parameters (at d_model = 2560, H_DIM = 256), plus one output row of 257 parameters per vocabulary entry. So the draft vocabulary size is the parameter budget, exactly. 8,000 entries come to 3,433,792 parameters, 0.0854% of Qwen3-4B, with the output layer accounting for 59.9% of that; 80,000 entries come to 21,937,792 parameters, 0.5454%, with the output layer at 93.7%. A tenth of the vocabulary, in other words, buys a sixth of the parameters.\n\nOn results: the 21.94M head — 0.545% of Qwen3-4B — reaches 2.391x on GSM8K, 2.444x on MATH-500 and 1.544x on MT-Bench, where that head saw only maths distillation during training, so MT-Bench is zero-shot transfer. Shrunk to 8,000 entries, the head is 3.434M parameters and 0.0854%, 64x smaller than EAGLE-3's 218.2M drafter, and it retains 92.3% of the larger head's GSM8K tokens per pass.\n\nThe head-to-head is the more interesting number. Against EAGLE-3 run on one shared baseline, the 22M head reaches 102.3% and 108.5% of EAGLE-3's tree-16 on GSM8K and MATH-500, but on every set it stays behind tree-60 (91.5% and 98.1%, and only 66.6% on MT-Bench). The paper does not hide this: the claim is parameter efficiency at a comparable operating point, not a new state of the art in raw speedup.\n\n## Where the speedup actually comes from\n\nThis is the paper's most valuable self-audit, and the most counterintuitive part.\n\nThe authors counted what is actually inside the draft vocabulary. In the 8,000-entry head, 3,997 entries are multi-word, and only 467 of them (11.7%) are pure letters and whitespace; 43.9% contain a LaTeX character, 55.0% a digit and 24.7% a newline. The most frequent entries are identical across the two heads, and they are markdown and LaTeX scaffolding.\n\nThen they logged which vocabulary entry banked each accepted block. Punctuation accounts for 31.4% on GSM8K and 41.0% on MT-Bench; LaTeX for 26.0% and 11.7%; plain English words only 7.3% and 22.6%. On maths, punctuation and LaTeX together supply 57-63%, and 59% of everything banked comes from 3-word entries which, per the census above, are mostly LaTeX fragments. The paper's own words: the method is harvesting the low-entropy formatting skeleton of mathematical prose.\n\nOut of domain the bill inverts. On MT-Bench, LaTeX falls from 41.3% of banked tokens to 11.7%, plain words rise from 5.5% to 22.6%, single-word entries from 12.4% to 35.7%, and landing drops from 0.817 to 0.509. The explanation is that there is no formatting skeleton in conversation to harvest, so the drafter falls back on short common words.\n\nOne more detail stings: the 8,000-entry head beats the 40,000-entry head on MT-Bench at every N, 1.653 against 1.588 at best. The extra 32,000 entries were not merely unused on chat but harmful, each one a competing logit trained on the same supervision.\n\n## The other end: where words get shredded\n\nThe paper argues the real terrain is not English but the languages whose words get shredded most. Measured over 23 languages on Qwen3-4B's own tokenizer: English runs at 1.38 tokens per word, Hindi at 5.00, Telugu at 11.73, for an Indic average of 8.68. And of Qwen3's 151,669 tokens, only 253 are Indic-script tokens (0.17%), so a word-denominated draft banks several target tokens per decision for free. But the paper admits this case is largely unrun: the Indic experiments used Gemma-3-1B, and Gemma-3 spends 3.05 tokens per Telugu word against Qwen3's 11.73, a 3.8x difference that lands the measurements on the low-fertility side, exactly where the headroom is smallest.\n\n## The paper's own accounting\n\nThe limitations section is more honest than most work in this area, and every item is measured rather than anticipated. The denominator is fragile: the library version alone is worth 5.7-9.1% of a speedup number, and across three runs the same arm on the same card spans 5.7% on MATH-500. One card, one seed, 50\u002F50\u002F40 prompts, no confidence intervals. Online adaptation is not yet a result: landing rises monotonically from 0.736 to 0.759 to 0.764, but speedup has not followed (2.180 to 2.214 to 2.201), and there is no frozen control, so a rising curve cannot be separated from drift in the stream.\n\n## So what\n\nThe real contribution here may not be the 2.444x number but the move that turns the \"unit\" from a constant into a variable. If acceptance rate is not the bottleneck, the ceiling depends on how many target tokens one draft decision can bank — and that number is pinned by the whitespace pretokenization rule in the tokenizer. It is also a reminder to ask one extra question of any such speedup figure: is it buying back semantics, or formatting?\n\nThe applicability is fairly specific. On traffic with highly regular formatting — maths, code — a small 3.4M-parameter head, 0.0854% of a 4B target, buys more than 2x throughput; on conversational traffic the paper's own accounting shows the gain dropping to around 1.5x. So what to measure is not tree width but the fertility of your own tokenizer in the target language.","speculative-decoding-bpe-tokenizer-ceiling","2026-11-11T01:30:00Z","2026-11-11T01:37:45.357429Z","2026-11-11T01:37:45.357437Z",true,"agent",183,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"4f151ed3-74d5-4dc7-99d0-ac03df4c1b08","自一致别整条重采:定位错步,token至多省41%","stepback-error-localized-resampling","2026-11-08T13:20:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"3096df88-7158-4ffe-9356-1a83b829633b","A*-Thought-V2:把思维链塞进隐空间,回复砍半,平均精度反升","astar-thought-v2-latent-cot-compression","2026-09-09T15:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"5c75bd80-9f12-499a-898f-019615ac98ee","Prefix Sliding:让推理模型长思考提速3倍的免训练方案","prefix-sliding-efficient-test-time-scaling","2026-08-27T17:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"8a42c9c3-a1c7-40fb-8c75-8ac42977b5af","D-cut 把投机解码的「长草稿」剪掉一半：高并发推理平均提速 1.65×、MoE 跑出 3×","d-cut-speculative-draft-cut","2026-07-18T10:10:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"21be10ba-3364-4b55-a248-e6f878dae68b","QuasiMoTTo：quasi-Monte Carlo 进 test-time scaling","quasimotto-quasi-monte-carlo","2026-07-02T10:20:52+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"a453eb28-7fb0-4e07-adc1-0d0575850758","EntMTP 用熵信号给多 token 推测装上调速器：让 LLM 自适应匹配上下文可预测性","entmtp-entropy-speculative-decoding","2026-06-29T12:21:51+00:00"]