[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tensorrt-edge-llm-0-10-multimodal-runtime":3,"news-related-ce70384a-990b-4994-bfb6-27775be45661":41},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"ce70384a-990b-4994-bfb6-27775be45661","TensorRT Edge-LLM 0.10.0：边端第一个统一的 C++ 多模态推理栈","TensorRT Edge-LLM 0.10.0 把 LLM\u002FVLM\u002F世界模型\u002F扩散文本\u002FASR 全部塞进同一个边缘端 C++ 运行时,并实验性开放免 ONNX 的引擎构建通道。","NVIDIA 边缘推理框架 TensorRT Edge-LLM 在 8 月 12 日发布的 0.10.0 版本里，把「Day-0 支持新模型」的口径往前推了一大截——不只是给一个新模型加个导出脚本，而是把多种完全异构的模态（LLM、VLM、世界模型、文本扩散、多语种 ASR）全部塞进同一个 C++ 运行时，跨 Jetson、DRIVE、DGX Spark 跑同一套引擎。这是边缘推理栈从「单模型优化器」往「物理 AI 通用推理基础」转向的一个清晰信号。\n\n## 一次发掉了五种工作负载的 Day-0 支持\n\n0.10.0 的 Key Features 列得很直接：Qwen3.8-27B Day-0 支持、Nemotron-3.5 Lightning（30B 总参 \u002F 3B 激活 MoE）的 MTP 与 DFlash 投机解码、Cosmos3-Edge 多模态推理（含实验性策略与动作生成）、DiffusionGemma 26B-A4B 的 NVFP4 块扩散推理、Nemotron-3.5-ASR 0.6B 流式 ASR，再加上 DSpark（带 DDTree drafting 的投机解码）。[1]\n\n每条对边缘端的意义不一样：Qwen3.8-27B Day-0 让中文为主的工业 Agent 拿到原生强化推理基底；Nemotron-3.5 Lightning NVFP4 让 30B MoE 真正能在 Jetson Thor 一类设备上跑；Cosmos3-Edge 把世界模型搬到产线机器人附近；DiffusionGemma NVFP4 把文本块扩散首度推到边缘；Nemotron-3.5-ASR 在 0.6B 体量下覆盖 40 语种流式识别。\n\n把这五件事放到同一个 release 里，意味着 NVIDIA 已经在执行「一个 C++ runtime = 一台边缘设备所有模型」的工程目标，而不是分多套栈去分别维护。\n\n## 免 ONNX 直接构建引擎的实验通道\n\n0.10.0 里我比较在意的是「an experimental direct TensorRT engine builder without ONNX export」，也叫 direct engine builder。[1] 它跳过 PyTorch → ONNX → TensorRT 这条传统导出链，直接从 Hugging Face checkpoint 构建 TensorRT 引擎，与正路共享同一套 C++ 部署运行时。\n\n实际好处有三层：一是 build 阶段不再需要临时占用 GPU 显存做 ONNX 中转，对单卡开发者更友好；二是依赖更干净，避免了 ONNX opset 漂移导致的兼容性问题；三是配合新加的「multi-turn KV-cache reuse」（LLM 的 paged KV-cache + VLM 的 media-aware KV reuse），多轮对话与多图理解在长上下文里能把 KV 真正留住，不用每轮重算。0.9.1 已经把 paged KV-cache prefill 与 paged XQA decode 打通，0.10.0 是把这套机制向多轮推进的关键版本。[2]\n\n## 投机解码生态一次填齐\n\n边缘推理对延迟敏感，投机解码是关键工具。0.10.0 把 NVIDIA 内部的几个路径一起外推：Nemotron-3.5 Lightning 的 MTP+DFlash 全套，Qwen3 \u002F Qwen3.5 走 DDTree drafting 的 DFlash 扩展，以及新加入的 DSpark。DFlash 的原始思路：把扩散式采样当作 draft 阶段，用 AR 主模型做 verify，靠并行展开多个候选 token 拿到约 3× 加速。DSpark 据其 spec 描述走的是 EAGLE-style 的 lightweight draft + 主模型 verify 路径，配合 DDTree（一种树形并行展开）来提升 acceptance rate。[2][3]\n\n把这几条路径并排放进同一 release，意味着 NVIDIA 在边缘端投机解码这件事上不再只押一种 draft 策略，而是看模型本身的 MoE 形态、是否 MTP、是否含视觉\u002F语音分支来灵活切换。\n\n## 一点观察\n\nTensorRT Edge-LLM 的 release cadence 已经从「季度大版本」缩到「月度小版本」+「关键模型 Day-0」。0.10.0 的覆盖面——同一天拉起 LLM、世界模型、扩散文本、ASR、投机解码五条战线——意味着边缘 AI 不再被「下一个权重能不能跑」卡脖子，而是被「C++ runtime 能不能吃下」卡脖子。这条赛道接下来的比拼点，是运行时能否跟得上基础模型每两周一发的节奏。开源生态做边缘端，NVIDIA 用 C++ runtime 抢到了关键卡位。\n\n---\n\n**参考**\n\n[1] NVIDIA TensorRT-Edge-LLM v0.10.0 Release Notes, 2026-08-12. https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-Edge-LLM\u002Freleases\u002Ftag\u002Fv0.10.0\n\n[2] NVIDIA TensorRT-Edge-LLM 项目 README. https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-Edge-LLM\n\n[3] Google Developers Blog: DFlash diffusion-style speculative decoding on TPU, 2026-05. https:\u002F\u002Fdevelopers.googleblog.com\u002Fsupercharging-llm-inference-on-google-tpus-achieving-3x-speedups-with-diffusion-style-speculative-decoding\u002F","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-Edge-LLM\u002Freleases\u002Ftag\u002Fv0.10.0","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"8dac812d-3839-4abe-a855-5f56ec9515fd","nvidia",{"id":25,"name":26,"slug":26,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"17e85010-71a8-4f10-a5c0-078c2f15ccd7","en","TensorRT Edge-LLM 0.10.0: edge's first unified C++ multimodal runtime","TensorRT Edge-LLM 0.10.0 packs LLM\u002FVLM\u002Fworld-model\u002Fdiffusion-text\u002FASR into one edge C++ runtime and opens an experimental ONNX-less engine builder.","NVIDIA's edge inference framework TensorRT Edge-LLM pushed the \"Day-0 model support\" bar up significantly in its 0.10.0 release on August 12. Rather than tacking on another export script for a single new model, it pushed every heterogeneous modality — LLMs, VLMs, world models, text diffusion, multilingual ASR — through the same C++ runtime, running identical engines across Jetson, DRIVE, and DGX Spark. This is a clear signal that the edge inference stack is shifting from \"single-model optimizer\" to \"general physical-AI inference substrate.\"\n\n## Day-0 support for five workloads in one release\n\nThe 0.10.0 Key Features are explicit: Day-0 support for Qwen3.8-27B; MTP and DFlash speculative decoding for Nemotron-3.5 Lightning (30B total \u002F 3B active MoE); multimodal reasoning with Cosmos3-Edge (including experimental policy\u002Faction generation); NVFP4 block-diffusion inference for DiffusionGemma 26B-A4B; streaming ASR from Nemotron-3.5-ASR 0.6B; plus DSpark (a speculative decoder with DDTree drafting).[1]\n\nEach item carries its own edge-side implication. Qwen3.8-27B Day-0 gives Chinese-leaning industrial agents a native reasoning base; Nemotron-3.5 Lightning NVFP4 lets 30B MoE actually fit on Jetson Thor-class devices; Cosmos3-Edge brings world models onto factory-floor robots; DiffusionGemma NVFP4 pushes text block-diffusion to the edge for the first time; and Nemotron-3.5-ASR streams recognition across about 40 languages in a 0.6B package.\n\nShipping all five together means NVIDIA is executing on an engineering target: \"one C++ runtime = every model on an edge device,\" instead of maintaining separate stacks per workload.\n\n## Experimental ONNX-less engine builder\n\nThe bit I find most interesting in 0.10.0 is \"an experimental direct TensorRT engine builder without ONNX export,\" also called the direct engine builder.[1] It skips the traditional PyTorch → ONNX → TensorRT export chain and builds TensorRT engines directly from Hugging Face checkpoints, sharing the same C++ deployment runtime as the conventional path.\n\nThere are three concrete benefits. First, the build phase no longer needs to allocate GPU memory for ONNX intermediate artifacts, which matters for single-GPU developers. Second, the dependency surface is cleaner, removing the ONNX opset drift that historically produced version-skew breakages. Third, paired with the newly added \"multi-turn KV-cache reuse\" (paged KV-cache for LLMs plus media-aware KV reuse for VLMs), multi-turn dialog and multi-image understanding can finally retain KV across long contexts without recomputing it each turn. 0.9.1 already enabled paged KV-cache prefill and paged XQA decode; 0.10.0 is the version that pushes that mechanism into multi-turn workflows.[2]\n\n## Speculative decoding lineup filled in at once\n\nEdge inference is latency-sensitive, and speculative decoding is the key tool. 0.10.0 unrolls several NVIDIA-internal paths simultaneously: the MTP+DFlash set on Nemotron-3.5 Lightning, the DDTree-drafting DFlash expansion for Qwen3 \u002F Qwen3.5, and the new DSpark path. The original DFlash idea uses diffusion-style sampling as the draft stage and verifies against an AR main model, parallelizing over candidate tokens for roughly 3× speedups. DSpark, by its spec, follows an EAGLE-style lightweight draft + main-model verify path, paired with DDTree (a tree-shaped parallel expansion) to lift the acceptance rate.[2][3]\n\nLining these up in the same release signals that NVIDIA no longer wants to back a single draft strategy at the edge; instead, it wants to switch strategies based on each model's MoE shape, whether it carries MTP, and whether it has vision or speech branches.\n\n## One observation\n\nTensorRT Edge-LLM's release cadence has shifted from \"quarterly majors\" to \"monthly minors plus Day-0 for key models.\" The breadth of 0.10.0 — booting LLM, world-model, diffusion-text, ASR, and speculative-decoding work in a single ship — means edge AI is no longer gated by \"can the next weights run here.\" It is gated by \"can the C++ runtime swallow them.\" The next competition in this lane is whether runtime engineering keeps up with base-model releases coming every two weeks. NVIDIA has claimed a critical position with its C++ runtime for the open-source edge ecosystem.\n\n---\n\n**References**\n\n[1] NVIDIA TensorRT-Edge-LLM v0.10.0 Release Notes, 2026-08-12. https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-Edge-LLM\u002Freleases\u002Ftag\u002Fv0.10.0\n\n[2] NVIDIA TensorRT-Edge-LLM project README. https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-Edge-LLM\n\n[3] Google Developers Blog: DFlash diffusion-style speculative decoding on TPU, 2026-05. https:\u002F\u002Fdevelopers.googleblog.com\u002Fsupercharging-llm-inference-on-google-tpus-achieving-3x-speedups-with-diffusion-style-speculative-decoding\u002F","tensorrt-edge-llm-0-10-multimodal-runtime","2026-08-23T00:00:00Z","2026-08-23T07:07:52.163784Z","2026-08-23T07:07:52.163799Z",true,"agent",49,{"items":42},[43,48,53,58,63,68],{"id":44,"title":45,"news_slug":46,"published_at":47},"a8b9d045-0f4c-4596-baa7-060955365877","TensorRT Edge-LLM 0.10.0：Qwen3.8-27B Day-0 上车，边缘 LLM 推理再加速","tensorrt-edge-llm-qwen3-8-27b-day0","2026-08-21T15:00:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"7bdf2df4-bb0f-42ae-b0ff-187de2e3558e","把语音 AI 拆成开源乐高：HF + Cerebras 用 Gemma 4 + Qwen3-TTS 拼出实时对话流水线","hf-cerebras-voice-ai","2026-07-02T02:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"68b6bbbc-384c-43fb-8024-2cf050107149","稀疏MoE+投机解码：开源模型首次在推理速度上超越闭源方案","stepfun-step-3-7-flash-409-tps-198b-moe","2026-06-04T13:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"4d436945-18e9-4d69-a4c8-c1e3e975ab33","MiniMax M3发布：稀疏注意力打通百万token上下文，开源模型编程能力逼近闭源前沿","MiniMax-m3-sparse-attn-million-token-msa","2026-06-04T01:00:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"63255594-b16a-4106-9acc-2dc479b97e14","英伟达Nemotron 3 Ultra登场：550B开源模型刷新美国开放权重智能榜单","nvidia-nemotron-3-ultra-550b-48-intelligence","2026-06-01T10:10:00+00:00",{"id":69,"title":70,"news_slug":71,"published_at":72},"2b5b7c66-7289-45db-b5b7-dea67882310c","NVIDIA 发布 Nemotron-Labs Diffusion：三模态语言模型统一 AR 与扩散解码","nvidia-nemotron-diffusion-ar-dllm-tri-modal","2026-05-23T04:10:00+00:00"]