With Coding Agents like Claude Code and Codex continuing to scale, the underlying inference engine is facing unprecedented pressure — long context, multi-turn dialogue, real-time response — and traditional general-purpose inference frameworks are stretched thin in these scenarios. In May 2026, the LightSeek Foundation released TokenSpeed, an open-source LLM inference engine purpose-built for agentic workloads, targeting TensorRT-LLM-class performance.
Why agentic inference is a different problem
Coding Agent requests differ fundamentally from ordinary chat: context routinely exceeds 50K tokens, and dialogues can last dozens of turns. This puts simultaneous pressure on two metrics: per-GPU TPM (tokens per minute), determining how many users a single card can serve; and per-user TPS (tokens per second), determining whether the response feels smooth. Most public benchmarks don't cover this kind of scenario.
TokenSpeed's goal is to maximize both metrics simultaneously: under a per-user TPS floor (typically 70 TPS, 200+ TPS in some scenarios), maximize per-GPU TPM.
The five-layer architecture's design logic
TokenSpeed's architecture is built around five core design pillars: a compiler-driven parallel modeling layer, a high-performance scheduler, safe KV-resource-reuse limits, a pluggable tiered kernel system, and SMG-integrated low-overhead CPU-side request ingress.
At the modeling layer, TokenSpeed adopts a native SPMD (Single Program Multiple Data) parallel mode — all processes run the same program but operate on different data subsets. Developers only need to mark I/O positions at module boundaries, and a lightweight static compiler automatically generates collective communication logic, eliminating the need for manual implementation.
The scheduling layer structurally separates the control plane from the execution plane. The control plane is implemented in C++ as a finite state machine, with the type system enforcing safe resource management (including KV cache state passing and lifecycle) at compile time, rather than relying on runtime conventions. The execution plane uses Python, preserving development efficiency.
The kernel layer treats GPU kernels as first-class modules rather than hardcoding them into the engine core, providing a portable public API, a centralized registration-and-selection model, and an extensible plugin mechanism that supports heterogeneous accelerators. The team has also developed one of the fastest MLA (Multi-head Latent Attention) kernels for NVIDIA Blackwell. Notably, TokenSpeed's MLA has been adopted by vLLM.
Implications for the industry
TokenSpeed's emergence reflects a new trend taking shape: inference engines are moving from "general-purpose efficiency" to "deep scenario optimization." As the industry shifts from "the bigger the model the better" to "the cheaper the inference the better," the competition among inference frameworks is also moving from raw performance numbers to refined polish on vertical scenarios. The agentic-inference battlefield is just getting started.