ThunderAgent: Scheduling Agentic LLM Inference at Program Granularity, Killing KV Cache Thrashing and Doubling Throughput

On July 29, 2026, the Together AI engineering team, in collaboration with Georgia Tech, UIUC, and CMU, announced ThunderAgent — a new scheduling framework purpose-built for agentic LLM inference. Code and paper (arXiv:2602.13692) were released in lockstep. The work has been accepted as an ICML 2026 Spotlight and squarely attacks the dirty secret of every mainstream inference engine running large-scale agent workloads today: KV cache thrashing.

Background: Why today's vLLM / SGLang / TensorRT-LLM engines over-promise under agent loads

The blog post frames the diagnosis in plain English. Every one of these engines is a request-level scheduler: every individual LLM call is treated as an atomic unit, with no visibility into the fact that it belongs to a longer-running agentic workflow.

A typical agentic workload oscillates between two phases:

  • a reasoning phase, where the GPU is busy churning out tokens;
  • an acting phase, where the model blocks on an external tool (compiler, search, shell, etc.) and the GPU is completely idle.

Run hundreds of such agents concurrently on an 8×H100 node and every agent's KV cache grows during each phase, competing for the same finite pool of GPU memory. Under memory pressure, the legacy scheduler panics and falls back to the obvious eviction policy: LRU. Agent A pauses waiting for a tool call, its KV cache gets evicted to make room for Agent B's prefill. Seconds later Agent A's tool returns, and the engine has to re-prefill Agent A's entire conversation history from scratch, which in turn boots Agent C's cache out. The cascade is what the authors call thrashing — and it is exactly what ThunderAgent takes its name from.

The baseline numbers in the paper are sobering. On 8×H100 with HiCache offloading and a batch size of 192, SGLang's default scheduler tops out at 390 token/s with a mean latency of 65 seconds.

How ThunderAgent fixes it: turning every workflow into a schedulable program

ThunderAgent is not a new inference engine. The novelty sits one layer above: in the granularity of scheduling decisions. It is a lightweight layer that lives between agentic clients and inference backends, and the primitive it schedules on is the program-level workflow, not the individual request.

Three concrete moves:

  1. Workflow abstraction as a program. The scheduler keeps a program table that tracks every agent workflow's current phase (reasoning or acting), its KV-cache footprint, and which node it currently lives on.
  2. Active pausing of low-priority programs. Under memory pressure, instead of blindly evicting cache, the scheduler pauses low-priority workflows entirely so the remaining active ones keep their KV cache hot.
  3. Global waiting queue + load balancing. Resumed programs are routed to whichever node currently has the most spare KV capacity, rather than being statically pinned. This avoids the memory imbalance that SGLang Gateway's session-based static pinning creates when agent context lengths get long-tailed.

Crucially, ThunderAgent composes with KV-cache offloading schemes like HiCache and LMCache rather than replacing them. It treats GPU HBM, CPU RAM, and disk as one unified cache pool, so offloading stops being a temporary patch and starts working in earnest alongside program-level scheduling.

What the numbers look like

Together AI ran its own internal CoderForge synthetic-data-generation pipeline (hundreds of coding agents in sandboxes, dozens of turns each) as the testbed:

  • Single node, batch 192: throughput goes from SGLang's 390 token/s to ThunderAgent's 803 token/s (≈ 2.06×), and mean latency drops from 65 s to 10.6 s (≈ 6×).
  • 8 nodes / 64 GPUs: throughput scales near-linearly from 671 steps/min at 16 GPUs to 2,248 steps/min. The speedup over SGLang Gateway widens from 1.79× at 2 nodes to 2.39× at 8 nodes.
  • Drop-in ergonomics: the only client-side change is adding a program_id field. The OpenAI-compatible interface and pass-through compatibility with existing optimizations (speculative decoding, quantization) are preserved.

The blog also notes that ThunderAgent has already been integrated into SkyRL and NVIDIA Dynamo — and Dynamo in particular is NVIDIA's big bet on disaggregated inference. The fact that it is willing to plug in a third-party scheduler at this layer suggests the program abstraction will land in production framework roadmaps sooner than later.

So what: is "the next Transformer moment" hiding in the scheduler?

For the past two years, inference-stack improvements have mostly been about "make one request go faster" — PagedAttention, continuous batching, prefix caching, FlashAttention. The agent era changes the shape of the workload from "one-shot Q&A" to "long, multi-turn, tool-using workflows", and the scheduler's window of visibility has to move from request up to program.

The single sentence that captures ThunderAgent is: once the scheduler can see that two adjacent LLM calls belong to the same agent, it can make smarter trade-offs than LRU ever will. Like Thompson Sampling or Speculative Decoding, it looks trivially obvious in hindsight — but the author affiliation stack (Georgia Tech / UIUC / CMU / Together AI) suggests that mainstream inference frameworks will absorb the abstraction quickly.

For builders of agent infrastructure, the immediate follow-up is not "write yet another inference engine" but "be the first to ship a clean, well-engineered implementation of program-level scheduling for multi-agent, long-horizon workloads" — especially in environments with non-NVIDIA accelerators and Chinese-language tooling, where public implementations are still sparse.

References: Together AI blog post • arXiv paper 2602.13692 • GitHub repo