[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tokenspeed-agentic-inference-engine":3,"topics-all":36,"news-related-068bcc7a-d901-4d3d-924c-eefa7ced4467":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"068bcc7a-d901-4d3d-924c-eefa7ced4467","TokenSpeed 开源推理引擎发布：剑指 Agentic Workloads 的高效推理","随着 Claude Code、Codex 等 Coding Agent 规模不断扩大，底层推理引擎正面临前所未有的压力——长上下文、多轮对话、实时响应，传统的通用推理框架在这些场景下左支右绌。LightSeek Foundation 于 2026 年 5 月发布了 TokenSpeed，一款专为 Agentic 工作负载设计的开源 LLM 推理引擎，剑指 TensorRT-LLM 级别性能。\n\n**为什么 Agentic 推理是另一个问题**\n\nCoding Agent 的请求与普通聊天截然不同：上下文动不动超过 50K token，对话往往持续数十轮。这对两个指标同时施压：一是每 GPU TPM（每分钟 token 数），决定单卡能服务多少用户；二是每用户 TPS（每秒 token 数），决定用户是否觉得响应流畅。大多数公开 benchmark 并不覆盖这类场景。\n\nTokenSpeed 的目标是同时最大化这两个指标：在保持每用户 TPS 底线（通常 70 TPS，部分场景 200+ TPS）的前提下，最大化 per-GPU TPM。\n\n**五层架构的设计逻辑**\n\nTokenSpeed 的架构围绕五个核心设计支柱展开：编译器驱动的并行建模层、高性能调度器、安全的 KV 资源复用限制、可插拔的分层内核系统，以及 SMG 集成的低开销 CPU 侧请求入口。\n\n在建模层，TokenSpeed 采用本地 SPMD（单程序多数据）并行模式——所有进程运行同一程序但操作不同数据子集，开发者只需在模块边界标注 I\u002FO 位置，轻量级静态编译器自动生成集合通信逻辑，省去手动实现的繁琐。\n\n调度层将控制平面与执行平面结构性分离。控制平面在 C++ 中实现为有限状态机，通过类型系统在编译时强制安全资源管理（包括 KV 缓存状态传递和生命周期），而非依赖运行时约定。执行平面则使用 Python 实现，保证开发效率。\n\n内核层将 GPU 内核视为一等模块而非硬编码到引擎核心，提供可移植的公共 API、集中式注册与选择模型，以及支持异构加速器的可扩展插件机制。团队还为 NVIDIA Blackwell 开发了目前最快的 MLA（Multi-head Latent Attention）内核之一。值得关注的是，TokenSpeed MLA 已被 vLLM 采纳。\n\n**对行业的启示**\n\nTokenSpeed 的出现折射出一个正在形成的新趋势：推理引擎正从「通用高效」走向「场景深度优化」。当行业从「模型越大越好」转向「推理越省越好」，推理框架的竞争也从性能数字游戏进入垂直场景的精细化打磨阶段。Agentic 推理的战场，才刚刚开始。","https:\u002F\u002Fwww.marktechpost.com\u002F2026\u002F05\u002F07\u002Flightseek-foundation-releases-tokenspeed-an-open-source-llm-inference-engine-targeting-tensorrt-llm-level-performance-for-agentic-workloads\u002F","8382d60c-c2c4-49c5-9638-8518b803f88f",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"07dd8e6c-8a8e-4e7a-9a4b-149cd1c76645","en","TokenSpeed: open inference engine for agent workloads","With Coding Agents like Claude Code and Codex continuing to scale, the underlying inference engine is facing unprecedented pressure — long context, multi-turn dialogue, real-time response — and traditional general-purpose inference frameworks are stretched thin in these scenarios. In May 2026, the LightSeek Foundation released TokenSpeed, an open-source LLM inference engine purpose-built for agentic workloads, targeting TensorRT-LLM-class performance.\n\n**Why agentic inference is a different problem**\n\nCoding Agent requests differ fundamentally from ordinary chat: context routinely exceeds 50K tokens, and dialogues can last dozens of turns. This puts simultaneous pressure on two metrics: per-GPU TPM (tokens per minute), determining how many users a single card can serve; and per-user TPS (tokens per second), determining whether the response feels smooth. Most public benchmarks don't cover this kind of scenario.\n\nTokenSpeed's goal is to maximize both metrics simultaneously: under a per-user TPS floor (typically 70 TPS, 200+ TPS in some scenarios), maximize per-GPU TPM.\n\n**The five-layer architecture's design logic**\n\nTokenSpeed's architecture is built around five core design pillars: a compiler-driven parallel modeling layer, a high-performance scheduler, safe KV-resource-reuse limits, a pluggable tiered kernel system, and SMG-integrated low-overhead CPU-side request ingress.\n\nAt the modeling layer, TokenSpeed adopts a native SPMD (Single Program Multiple Data) parallel mode — all processes run the same program but operate on different data subsets. Developers only need to mark I\u002FO positions at module boundaries, and a lightweight static compiler automatically generates collective communication logic, eliminating the need for manual implementation.\n\nThe scheduling layer structurally separates the control plane from the execution plane. The control plane is implemented in C++ as a finite state machine, with the type system enforcing safe resource management (including KV cache state passing and lifecycle) at compile time, rather than relying on runtime conventions. The execution plane uses Python, preserving development efficiency.\n\nThe kernel layer treats GPU kernels as first-class modules rather than hardcoding them into the engine core, providing a portable public API, a centralized registration-and-selection model, and an extensible plugin mechanism that supports heterogeneous accelerators. The team has also developed one of the fastest MLA (Multi-head Latent Attention) kernels for NVIDIA Blackwell. Notably, TokenSpeed's MLA has been adopted by vLLM.\n\n**Implications for the industry**\n\nTokenSpeed's emergence reflects a new trend taking shape: inference engines are moving from \"general-purpose efficiency\" to \"deep scenario optimization.\" As the industry shifts from \"the bigger the model the better\" to \"the cheaper the inference the better,\" the competition among inference frameworks is also moving from raw performance numbers to refined polish on vertical scenarios. The agentic-inference battlefield is just getting started.","tokenspeed-agentic-inference-engine","2026-05-20T19:10:00Z","2026-05-20T19:06:36.891522Z","2026-08-19T02:08:40.142862Z",true,"agent",337,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"1edd86ea-3eb0-4424-9f57-add15c08d891","HF Hub 接入 SkyPilot：Xet 去重让 20+ 云共享模型数据","hf-skypilot-hf-storage","2026-07-12T06:30:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"f76e6f4b-1ae5-442a-abfa-823e7226ae81","Hugging Face 把 transformers 跑出 vLLM 原生速度：单 flag 让 235B MoE 直接吃到 EP 红利","hf-vllm-transformers-backend","2026-07-09T04:03:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"4d436945-18e9-4d69-a4c8-c1e3e975ab33","MiniMax M3发布：稀疏注意力打通百万token上下文，开源模型编程能力逼近闭源前沿","minimax-m3-sparse-attn-million-token-msa","2026-06-04T01:00:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"86c3465b-27f0-4f8c-83d1-9a8e1fa171c8","Compute Aligned Training：让模型学会协同推理的新训练范式","compute-aligned-training-pass-n-vote-michigan","2026-05-22T13:10:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"0565190a-0bcd-492f-934f-0ad2ab32f485","70万参数2.8MB填一张表:Cua开源CUA-S1,单次前向替代23轮LLM","cua-s1-forms-system-one-model","2026-09-20T13:11:48+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"b571067a-9fa8-42bf-9431-98f26ac78e03","伯克利把LLM推理搬进SSD:KV缓存压缩15倍","llm-inference-in-flash-cim-ssd","2026-09-19T21:10:00+00:00"]