[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-gemma-4-assistant-speculative-3x":3,"topics-all":36,"news-related-3e8cae68-d18b-4168-b222-42f9acf30102":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"3e8cae68-d18b-4168-b222-42f9acf30102","Transformers 5.8.0：Gemma 4 Assistant 推测解码提速 3 倍","Hugging Face 于 5 月 13 日发布的 Transformers v5.8.0，引入了 Google 的 Gemma 4 Assistant 模型。这是一款专为推测解码（Speculative Decoding）设计的小型辅助模型，可令 Gemma 4 在本地推理时实现高达 3 倍的解码加速，同时完全复用主模型的 KV Cache，省去预填充（Pre-fill）阶段的开销。\n\n标准自回归语言模型每次前向传播只生成一个 token，是推理延迟的主要瓶颈。推测解码的基本思路是：训练一个「草稿模型」快速预测多个后续 token，再由主模型验证其中哪些正确。Gemma 4 Assistant 的创新在于，它与主模型 Gemma 4 共享同一个 Transformer 骨架，但额外引入跨注意力层，使其能够直接复用主模型在推理时已经填充的 KV Cache。这意味着辅助模型可以「借用」主模型的上下文表示，实现更准确的预测。\n\n关键架构细节包括：KV 共享——整个模型通过 KV 共享机制复用主模型的注意力状态，无需独立填充，节省约 30-50% 的推理时间；多 token 预测——每次草稿轮次可预测多个候选 token；跳过预填充——由于 KV 已就位，首个 token 的计算量大幅降低，对长上下文场景尤为友好。\n\n此前，推测解码的常见实现是训练一个独立的、更小的草稿模型（如 Medusa），但这类方法面临两个问题：一是草稿模型与主模型架构不同，无法共享 KV Cache；二是两个模型的输出分布可能存在偏移，导致接受率不稳定。Gemma 4 Assistant 采用了共享骨架加专用交叉注意力的方案，让辅助模型和主模型形成紧密耦合的「推测-验证」闭环，而非各自为战。\n\n对本地部署而言，这意味着在同等硬件上，可以用更低的延迟跑动 Gemma 4 34B 或 27B 等较大的开源模型。对边缘设备来说，在内存带宽受限的环境下，减少约 1\u002F3 的 token 生成时间，可能就是「能用」和「卡死」的区别。\n\nGemma 4 Assistant 目前已随 Transformers v5.8.0 正式合并，用户可以通过 AutoModelForCausalLM 加载并与 Gemma 4 主模型配合使用。从产业角度看，Google 正在构建一套围绕 Gemma 的完整推理优化工具链：MTP 推测解码、蒸馏压缩——这些技术组合起来，让开源小模型在端侧的生产可用性大幅提升。Hugging Face 将其纳入官方 release，意味着这套方案正在成为开源社区的标准实践。","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Freleases\u002Ftag\u002Fv5.8.0","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"8cf7490f-2449-4ba7-be19-61befa0d92b4","google",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"54796d2b-a3d5-4a85-a410-6f33e67adb71","en","Transformers 5.8.0: Gemma 4 Assistant speeds local inference 3x","Hugging Face's Transformers v5.8.0, released on May 13, introduces Google's Gemma 4 Assistant model. This is a small auxiliary model designed for speculative decoding, enabling up to 3× decoding speedup for Gemma 4 during local inference, while completely reusing the main model's KV cache, eliminating the pre-fill phase overhead.\n\nStandard autoregressive language models generate only one token per forward pass, the main bottleneck for inference latency. The basic idea of speculative decoding: train a \"draft model\" to quickly predict multiple subsequent tokens, then have the main model verify which are correct. Gemma 4 Assistant's innovation is that it shares the same Transformer backbone as the main model Gemma 4, but additionally introduces cross-attention layers, allowing it to directly reuse the main model's already-filled KV cache during inference. This means the auxiliary model can \"borrow\" the main model's contextual representation for more accurate predictions.\n\nKey architectural details include: KV sharing — the entire model reuses the main model's attention state via KV sharing, no independent fill needed, saving roughly 30-50% of inference time; multi-token prediction — multiple candidate tokens can be predicted per draft round; skip pre-fill — since KV is already in place, the first token's compute load drops dramatically, especially friendly for long-context scenarios.\n\nPreviously, common implementations of speculative decoding trained an independent, smaller draft model (like Medusa), but this faced two problems: first, the draft model and main model had different architectures, unable to share KV cache; second, output distributions between the two models could shift, leading to unstable acceptance rates. Gemma 4 Assistant uses a shared-backbone-plus-dedicated-cross-attention scheme, letting the auxiliary model and main model form a tightly-coupled \"speculation-verification\" closed loop, rather than each operating independently.\n\nFor local deployment, this means on the same hardware, larger open-source models like Gemma 4 34B or 27B can run with lower latency. For edge devices, in memory-bandwidth-constrained environments, reducing generation time by about a third could be the difference between \"usable\" and \"stuck.\"\n\nGemma 4 Assistant is now officially merged with Transformers v5.8.0, and users can load it via AutoModelForCausalLM to work with the Gemma 4 main model. From an industry perspective, Google is building a complete inference-optimization toolchain around Gemma: MTP speculative decoding, distillation compression — these technologies combined significantly improve the production usability of open-source small models on the edge. Hugging Face including it in an official release means this solution is becoming standard practice in the open-source community.","gemma-4-assistant-speculative-3x","2026-05-14T22:05:00Z","2026-05-14T22:06:43.341867Z","2026-08-19T02:08:40.142862Z",true,"agent",282,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"02c96622-1409-4867-9f25-772936edc6cd","滑动窗口反超SnapKV:NVIDIA重审投机解码KV压缩","rift-draft-kv-cache-compression","2026-10-23T15:20:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"4d36e54d-ca37-4d34-b8d9-74ebc25762cf","PACE:递归状态量化损失主因是舍入频率","pace-four-bit-recurrent-state-quantization","2026-10-23T01:10:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"be76fca7-bc02-4a76-a58f-17a661b82b60","SparPR:消费级显卡的KV缓存回收术","sparpr-kv-cache-recovery-consumer-gpu","2026-10-22T01:20:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"8c579d8d-4114-4d7d-92c1-22f5fab0537b","连续流语言模型跑到 8B：Lafa 单模型自验证解码提速 2.54 倍","large-flow-language-models-lafa-self-speculative-decoding","2026-10-21T03:09:03+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"8c6d408b-2913-48b2-bc1f-4e958ad854c5","Space Bunny 谜底揭晓:MiniMax M3.1 Flash,匿名期吞下 200 万亿 token","minimax-m3-1-flash-space-bunny","2026-10-19T21:10:12+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"559df680-f13e-48f0-9263-73116aaff411","上交大DLite免历史投机解码:Agent提速2.74倍","dlite-history-free-speculative-decoding","2026-10-19T17:07:52+00:00"]