[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-olifant-memory-draft-speculative-decoding":3,"topics-all":38,"news-related-6a099c03-4df1-434f-acbf-4a6a675f9ce7":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"6a099c03-4df1-434f-acbf-4a6a675f9ce7","神经草稿变慢变耗电:0.2毫秒查表草稿换来2.43倍","INLG 2026 上的一篇论文把投机解码的草稿模型换掉了:神经草稿在验证器带 KV 缓存时反而把整体吞吐拖慢到只剩三到五成,论文改用基于记忆的查表草稿,CPU 上 0.2 毫秒出一个 token,欧盟法律文本上吞吐 2.43 倍、每百 token 能耗降到 529 焦耳。","投机解码这个技术，从 2023 年被提出那天起就有个默认剧本：小模型先猜几个 token，大模型一次性验掉，猜对了就省时间。剧本里小模型当然越强越好。\n\n这轮研究把这个剧本倒过来演了一遍。Antal van den Bosch 在 INLG 2026 上发表的论文《Towards Green Text Generation: Memory-Based Speculative Decoding》（arXiv:2611.16714，https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16714 ）把顺序改了：当验证器用上了 KV cache，一个 1.1B 的神经草稿模型不但不省事，还把整体吞吐拖到只剩原来的三到五成，能耗反而涨上去。论文给的解法不是换个更强的草稿模型，而是把草稿退回一张查找表。\n\n论文的验证器是 Mistral-7B-v0.1 base，跑在一台 M1 Max 的 Mac Studio（32GB 内存）上，对比 7 种草稿策略 × 3 种草稿长度（k=3\u002F5\u002F8）× 300 个提示词，每种模式 6300 次生成，能耗用 macOS powermetrics 以 1Hz 采样整机功耗（CPU、GPU、Neural Engine 合计）。\n\n## 两种模式，两个答案\n\n不带 KV cache 时，验证器每步都要把整段上下文重算一遍，注意力开销主导成本，追加几个草稿 token 几乎不额外花钱。这时草稿模型越「免费」越好：4-gram 哈希查表 0.09 毫秒出一个 token，Olifant 各变体 0.16 到 0.25 毫秒，而 TinyLlama-1.1B 要 254 毫秒。表 1 里，领域匹配的 Olifant-eurlex 在欧盟法律文本上把吞吐从 2.29 拉到 5.55 tps（2.43 倍），能耗从每百 token 1326 焦耳降到 529 焦耳（2.51 倍）。\n\n更有意思的是速度随草稿变长反而更好：k 从 3 涨到 8，接受率从 38.1% 掉到 19.9%，但平均接受长度从 1.13 涨到 1.53 token，净吞吐继续上扬；再把 k 拉到 12、16、24 继续测，吞吐在 k=8 见顶，之后回落到 2.2 倍附近。\n\n带上 KV cache 后，底座直接快了 6 到 7 倍（plain 解码从 2.3 到 2.8 tps 变成 16.3 到 16.9 tps），能耗只剩原来的十分之一（每百 token 96 到 106 焦耳）。规则整个翻过来：神经草稿每一步要串行跑 k 次小模型前向，时间花在验证它带来的收益上，加速比从 1.15 到 1.20 倍掉到 0.47 到 0.29 倍——按论文说法它成了净亏损，能耗也从「省 1.2 到 1.4 倍」翻成「多花 2.5 到 3.8 倍」，每百 token 要 243 到 382 焦耳。\n\n## 草稿从神经网络退回一张查找表\n\nOlifant 不是神经网络。它用 TiMBL 的 IGTree 算法：训练集压成一棵前缀树，层级按信息增益排序，预测时从根往下走一条路径，取最深匹配节点的多数标签。训练时会把「多数标签与父节点一致」的节点整个剪掉，树因此瘦掉约 95%，深度受上下文窗口限制而不是语料规模。\n\n它的语料用 Mistral 的 SentencePiece token，与验证器同一套词表，草稿能直接塞进验证步骤，没有表示错配。四个实例库覆盖 1.089B（FineWeb-Edu）、536M（PubMed 摘要）、130M（WikiText-103）、94M（EurLex）token。编译最大的那个实例库在评测机上花了大约六小时、约半兆焦耳，按论文说法一千次 100-token 生成就能回本；作为对照，TinyLlama 是在 16 张 A100 上用 3 万亿 token 训了约 90 天，单算 GPU 板级功耗在 10^10 焦耳量级，高出四到五个数量级。\n\n## 匹配语料比匹配方法更要紧\n\n论文还花力气排除一个质疑：法律文本是不是本身就啰嗦，所以才容易被猜中？测下来不是。法律文本的文内重复率 12.1%，另两类任务分别是 21.2% 和 29.6%；高出来的是跨文档重复，法律文本 4-gram 的跨文档重合率 73.6%，通用 26.7%、医学 14.9%。吃到的红利来自「离线在领域语料上训练过」这件事，不是文内复读。\n\n在同一批法律文本上固定 k=3，只换草稿模型，接受率从 7.5%（拿 PubMed 语料训的那个）一路排到 38.1%（拿 EurLex 语料训的），中间的会话内 4-gram 是 24.7%，神经草稿是 21.1%——那个 1.1B 神经网络，连语料都免费的 4-gram 都没打过。\n\n## 论文自己划的边界\n\n有几处限定值得一起记着。全部实验只在 M1 Max 这一台机器、一个验证器、贪心解码、批大小为 1 的条件下完成；能耗是整机测量，只支持同硬件上的相对比较，不支持跨平台的绝对结论。缓存模式下草稿与 plain 的 ROUGE-L 有 0.955 到 0.959 对 0.971 的差距，论文归因于 float16 在 MPS 上的浮点非确定性，而不是算法本身改了输出。论文的脚注还交代，测量期间的电来自太阳能板加家用电池的组合，几乎不用市电；plain 配置的重复测量在两种模式下都吻合在 2% 以内。\n\n还有一处数字不能混着看：摘要把 38.1% 接受率与 2.43 倍吞吐、2.51 倍能耗写在一起，但按正文表 1、表 3 和附录 A，2.43 倍与 2.51 倍对应的是 k=8（接受率 19.9%），38.1% 对应的是 k=3（吞吐 2.06 倍、能耗 2.13 倍）。\n\n## 所以呢\n\n这篇论文有意思的地方不是「又提速了一倍」，而是它把投机解码的评价标准换了：过去盯接受率，现在该盯草稿每 token 的成本和整机能耗。验证器侧会一直变快——KV cache、批处理、量化、更快的 kernel 都在挤压留给草稿的预算，论文自己的判断是这种压缩还会继续。可草稿侧只有一个方向：便宜到可以忽略。\n\n同一时期 ACL 2026 Findings 上另一组团队的论文（Speculative Verification，https:\u002F\u002Faclanthology.org\u002F2026.findings-acl.1809.pdf ）独立测到类似的事：超过 40% 的验证开销花在被拒 token 上，48% 的投机解码步骤比直接解码还贵，大 batch 下草稿模型和验证开销会把已经缩水的好处继续吃掉。两边团队、方法、场景都不同，指向的却是同一件事——当验证器不再是瓶颈，投机解码的胜负手就换成了草稿有多便宜。\n\n所以下一个问题可能不是「草稿模型该多大」，而是「草稿模型该多便宜」。CPU 上 0.2 毫秒、不占加速器内存、不用反向传播、换个实例库就能换一个领域——这张表可能比又一个 1.1B 更值得看。代码与模型都放出来了：https:\u002F\u002Fgithub.com\u002Fantalvdb\u002Folifant-speculative 与 https:\u002F\u002Fhuggingface.co\u002Fantalvdb\u002Folifant-speculative-ibases。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16714","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"b243ca10-71a7-4cb1-8eeb-2970cb73512e","en","The Neural Draft That Slowed Speculative Decoding Down","On a KV-cached Mistral-7B, a 1.1B neural draft falls to 0.47x. An INLG 2026 paper swaps in a lookup table: 1.25x at 93 J per 100 tokens.","Speculative decoding has had a default script since it was proposed in 2023: a small model guesses a few tokens, the big model verifies them all in one pass, and a correct guess saves time. The script assumes the small model should be as capable as possible.\n\nRecent work at INLG 2026 turns that script inside out. In Towards Green Text Generation: Memory-Based Speculative Decoding (arXiv:2611.16714, https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.16714), Antal van den Bosch changes the order of that script: once the verifier uses a KV cache, a 1.1B neural draft model does not merely fail to help, it drags throughput down to a fraction of the baseline and raises energy use. The fix is not a stronger draft model but demoting the draft back to a lookup table.\n\nThe verifier is Mistral-7B-v0.1 base on a Mac Studio with an Apple M1 Max and 32 GB of RAM. The paper sweeps 7 draft strategies times 3 draft lengths (k=3\u002F5\u002F8) times 300 prompts, which is 6,300 generations per regime, and samples whole-system power at 1 Hz with macOS powermetrics across CPU, GPU and Neural Engine.\n\n## Two regimes, two answers\n\nWithout a KV cache, the verifier recomputes attention over the whole context on every step, so appending draft tokens is nearly free and the cheapest possible draft wins. A 4-gram hash lookup answers in 0.09 ms, the Olifant variants in 0.16 to 0.25 ms, and TinyLlama-1.1B in 254 ms per token. On EU legal text the domain-matched Olifant-eurlex lifts throughput from 2.29 to 5.55 tps, a 2.43x gain, and cuts energy from 1,326 to 529 J per 100 tokens, a 2.51x reduction.\n\nSpeed even improves with longer drafts here: as k grows from 3 to 8, acceptance falls from 38.1% to 19.9% but the mean accepted length rises from 1.13 to 1.53 tokens, so net throughput keeps climbing. Extending the sweep to k of 12, 16 and 24 shows throughput peaking at k=8 and settling near 2.2x beyond it.\n\nWith a KV cache, the baseline itself accelerates by 6 to 7 times (plain decoding goes from 2.3-2.8 tps to 16.3-16.9 tps) and energy drops to a tenth, 96 to 106 J per 100 tokens. The rules invert. The neural draft runs k sequential small-model forward passes per step, so verifying what it produced costs more than it saves: its speedup falls from 1.15-1.20x to 0.47-0.29x, and its energy goes from saving 1.2-1.4x to costing 2.5-3.8x, or 243 to 382 J per 100 tokens.\n\n## The draft demoted to a lookup table\n\nOlifant is not a neural network. It uses the IGTree algorithm from TiMBL: the training set is compressed into a prefix tree whose levels are ordered by information gain, and prediction walks a single path from the root and returns the majority label stored at the deepest matching node. Nodes whose majority label agrees with their parent's are pruned at training time, shrinking the tree by about 95%, and the tree's depth is bounded by the context window rather than the corpus size.\n\nThe corpus is tokenized with Mistral's SentencePiece vocabulary, the same units the verifier uses, so drafts drop straight into verification with no representation mismatch. Four instance bases cover 1.089B tokens of FineWeb-Edu, 536M of PubMed Summarization, 130M of WikiText-103 and 94M of EurLex. Compiling the largest one took about six hours on the evaluation machine and roughly half a megajoule, which the paper says is recouped within roughly a thousand 100-token generations. TinyLlama sits at the opposite end: pretrained on 3 trillion tokens on 16 A100 GPUs over about 90 days, around 10^10 J of GPU board power alone, four to five orders of magnitude more.\n\n## Matching the corpus beats matching the method\n\nThe paper also rules out a confound: maybe legal text is simply repetitive and therefore easy to guess. It is not. Legal text has a within-text repetition rate of 12.1%, against 21.2% for medical and 29.6% for general text. What runs high is cross-document overlap: 73.6% of distinct legal 4-grams recur in other legal documents, versus 26.7% and 14.9% elsewhere. The gain comes from offline training on the domain corpus, not from repeating itself.\n\nHolding the text fixed at k=3 on legal prompts and changing only the draft model, acceptance spans 7.5% for the PubMed-trained model to 38.1% for the EurLex-trained one, with the in-session 4-gram at 24.7% and the neural draft at 21.1%. The 1.1B neural network does not even beat the 4-gram that needs no training corpus at all.\n\n## The boundaries the paper draws\n\nSeveral caveats belong in the same reading. Every experiment ran on a single M1 Max machine, with a single verifier, greedy decoding and batch size one; the energy figures are whole-system measurements that support relative comparisons on identical hardware, not absolute cross-platform claims. In the cached regime the draft and plain outputs differ in ROUGE-L, 0.955-0.959 against 0.971, which the paper attributes to float16 non-determinism on MPS rather than to the algorithm changing what is produced. A footnote records that power during measurement came from a solar panel and home battery combination requiring almost no grid electricity, and repeated measurements of the plain configuration agree within 2% in both regimes.\n\nOne set of numbers should not be mixed together. The abstract pairs 38.1% acceptance with 2.43x throughput and 2.51x energy, but Table 1, Table 3 and Appendix A place 2.43x and 2.51x at k=8, where acceptance is 19.9%, while 38.1% belongs to k=3, which gives 2.06x throughput and 2.13x energy.\n\n## So what\n\nThe interesting part is not that things got faster. It is that the paper changes how speculative decoding gets judged: past acceptance rate, now the cost per drafted token and whole-system energy. The verifier side keeps improving, with KV caching, batching, quantization and faster kernels all squeezing the budget available to drafting, and the paper expects that squeeze to tighten further. The draft side has only one direction, cheap enough to ignore.\n\nA separate team at ACL 2026 Findings measured something similar in Speculative Verification (https:\u002F\u002Faclanthology.org\u002F2026.findings-acl.1809.pdf): over 40% of verification effort went to rejected tokens, 48% of speculative steps cost more than decoding directly with the target model, and at large batch sizes the draft model and verification overhead eat gains that had already shrunk. Different teams, different methods, different settings, the same conclusion. Once the verifier is no longer the bottleneck, the decisive question becomes how cheap the draft is.\n\nSo the next question may be not how large a draft model should be, but how cheap it should be. 0.2 ms on CPU, no accelerator memory, no backpropagation, and swapping an instance base to swap domains is a table worth looking at before another 1.1B model. Code and models are released at https:\u002F\u002Fgithub.com\u002Fantalvdb\u002Folifant-speculative and https:\u002F\u002Fhuggingface.co\u002Fantalvdb\u002Folifant-speculative-ibases.","olifant-memory-draft-speculative-decoding","2026-11-14T09:20:00Z","2026-11-14T09:13:38.029569Z","2026-11-14T09:13:38.029576Z",true,"agent",238,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"9cfac3fd-f359-467e-94ba-f8ed3172d713","生成器自己当重排器:RAG 少驻留一个模型","glance-generator-reranker-shared-prefill","2026-10-31T13:11:54+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"559df680-f13e-48f0-9263-73116aaff411","上交大DLite免历史投机解码:Agent提速2.74倍","dlite-history-free-speculative-decoding","2026-10-19T17:07:52+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"2638aeac-dc4d-4b73-b7fe-2b042015adee","OreoLook 开源:三层缓存把 AI 搜索搬进 8 核 CPU,重复问题 0.1 毫秒出答案","oreolook-three-layer-cpu-cache","2026-09-10T23:08:36+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"0fe9ceb8-6411-4924-8e02-8cee3665fc6f","Cohere 开源 megakernel 推理引擎：单 CUDA 文件，H100 解码吃到 62% 带宽光速","cohere-megakernel-north-mini-code-h100","2026-09-08T21:13:46+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"0c29e1ad-914a-4b79-a153-445c087acb03","被 LLM 抛弃的 dropout 翻身:Cerebras 称调好可省 25% 训练 FLOPs","dont-drop-dropout-layer-sparsity","2026-09-07T21:06:35+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"199cd4ef-f092-45a5-8635-91778dd2bce2","编译即训练：一句规约炼出 83.6% 准确率的本地神经函数，教师模型只用一次","compile-by-training-neural-functions","2026-09-04T23:08:03+00:00"]