When an agent session runs on a cluster, the GPU is not computing most of the time. More often it is waiting — waiting for a compiler, a retriever, or a shell command to finish. That waiting time is exactly the variable that decides how fast the whole system runs.
A team from Shanghai Jiao Tong University and Stanford took another run at that accounting. The paper TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving (arXiv:2610.37230, submitted October 27) treats "when will the tool come back" as a third scheduling signal, on a par with KV cache hits and token footprint, and then uses dynamic programming to jointly decide which sessions to admit and which node to place them on. In the authors' own evaluation, one configuration — DeepSeek-V4 Flash on SWE-bench Lite at 64 workers — shows latency down 26.8%, mean time to first token (TTFT) down 36.3%, and decode throughput up 29.5%.
The bill existing schedulers miss
The paper calls this scenario CGDC: GPUs clustered for inference, tool execution scattered across CPUs, with a single session hopping back and forth between the two. That makes a session's demand for GPU capacity interruptible — while a tool is running, the session is not holding the GPU.
Existing systems mostly follow an evaluate-then-decide pattern. Evaluation looks at GPU-side session features (whether the KV prefix can be reused, how many tokens will be burned), and a later stage decides admission and placement. The problem is that "when will the tool come back" is simply not part of that evaluation. A scheduler that cannot see tool-return timing is effectively ranking sessions against an incomplete valuation.
TAO's fix is to assign each session-node pair a triplet valuation: cache credit with time-aware decay, a binary readiness signal, and token count, compared lexicographically. Cache credit decays as a negative power of two with a 10-second half-life — the longer ago a reusable token was observed, the harder it is discounted. The readiness signal comes from a tool-duration predictor that prefers direct timing evidence, falls back to per-tool history or global history, and finally to a 5-second default. Which sessions to take and where to put them are solved in one shot by a capacity-state dynamic program, which revalidates against live state before applying.
The brightest cell and the darkest one
The headline numbers come from 12 model-workload-concurrency configurations. Sort the twelve latency improvements in Table 1 and the median is only 4.6%, with six of them at or below 5%. The 26.8% cell comes from DeepSeek-V4 Flash on SWE-bench Lite at 64 workers.
The darkest cell is more interesting. Table 3 tests workload compatibility on mixed and chatbot scenarios. Terminal-Bench mixed with SWE-Lite over 389 tasks shows TAO improving latency by 12.2%, 23.2% and 13.8% across the three concurrency levels. But the AutomationBench row of 800 tasks at 128 workers goes the other way: latency is negative 7.0%, TTFT is negative 504.0% (from Dynamo's 3.166 seconds up to TAO's 19.123 seconds), and decode throughput is negative 11.2%. Across the whole table, 5 of 36 cells are negative.
That is not necessarily a bug. A tool-readiness signal is meaningless for a pure chat workload — with no tool to wait for, it is noise, and it may actively pull scheduling off course. The paper puts this in the compatibility section rather than the abstract, which is fair. But the correct reading of "latency reduced by up to 26.8%" is "best case", not "typical case".
Which component is doing the work
Table 4 is the ablation, run on DeepSeek-V4 Flash, SWE-Lite, 32 workers. The full configuration decodes at 1417.97 tok/s with 96.31% prefix-cache hit rate and 0.154 seconds mean TTFT. Removing pieces one at a time:
- without cache credit: 1398.35 tok/s
- without readiness: 1363.06 tok/s
- without session evaluation: 1355.20 tok/s
- without decision making (DP replaced by best-fit-decreasing): 1233.72 tok/s
The last one hurts most — the decision side is worth more than the evaluation side. The hard part is deciding what to take and where to put it simultaneously under capacity constraints, and that is exactly where heuristic methods tend to degrade.
Section 4.3 breaks it down further. In a fixed replay of 218 tasks, cache-token ordering raises the prefix-cache hit rate by 0.19 percentage points over ThunderAgent and cuts mean pause wait by 4.5%; adding cache-credit decay lifts the hit rate by another 0.31 points and decode throughput by 1.68%. Turning on the readiness signal adds 4.03% decode throughput and cuts tool-return-to-first-token delay by 10.88%, with mean pause wait falling from 1.067 to 0.972 seconds.
Prediction accuracy is not the point
This is the part of the paper I think is most underrated. Appendix G runs an oracle experiment: feed the predictor the true tool duration from the replay. Prediction error drops by 99.53%–99.69%, down to 0.025–0.030 seconds.
But the payoff? Makespan improvement lands between −0.10% and +2.24%, and mean session latency between −0.36% and +0.71%. Essentially nothing.
What does matter is a different number: the longer the tool delay, the more DP placement decisions the readiness signal touches. At scale 1 (an extra 1–5 seconds per tool return) it changes 11 of 668 decisions, or 1.65%; at scale 2, 53 of 1345, or 3.94%; at scale 4 (an extra 4–20 seconds), 200 of 2521, or 7.93%. The slower the tool, the more valuable the judgment "don't grab the GPU yet, save it for the session about to come back." So TAO's gain does not come from predicting better — it comes from wiring an ignored dimension into the decision structure.
The cost is in Appendix I: across four-replica Qwen runs at 96–160 workers, planning submissions rise from 98 to 1,681, and at 160 workers a completed assignment solve averages 1.59 ms with a P95 of 3.68 ms, the two measured parts together accounting for 4.03 seconds, or 0.112% of the run. The DP search also has a fallback: 5.24% of submissions at 64 DeepSeek workers fell back to best-fit-decreasing, and the Qwen sweep ranged from 3.06% to 8.98%.
My read
The direction is right. The difference between last-generation LLM inference serving and agent infrastructure is precisely this: the old problem was "make one request fast," the new one is "keep the whole queue ordered while computation and waiting alternate." Leaving tool time out of the valuation means running an agent service as if it were an LLM service that never pauses — and that is a real blind spot for plenty of self-hosted agent platforms today.
But do not treat the paper's numbers as a promise. At the median of the same configuration set, latency gains are 4.6%, TTFT median 14.75%, decode throughput median 4.35%; only when the load is highly repetitive and concurrency tops a particular band do you reach 26.8%. The failure region is clear too: pure chat workloads, where at high concurrency TTFT can be an order of magnitude worse, and the single-node low-concurrency band, where TAO did not beat vLLM either. A different model tells a different story — on GLM-4.7 with SWE-verified, the paper reports latency down 16.2%–18.9%, TTFT down 15.7%–38.4%, and decode throughput up 22.8%–24.7%.
One more signal hides in Appendix E.2: when paired with an engine-level KV manager like TokenCake, adding cache-awareness alone is actually worse than using TokenCake by itself (at 16 workers, 137.907 seconds of latency versus 126.891), while TAO + TokenCake gives 122.940. Scheduling-layer judgment and engine-layer cache management do not simply compose; the wrong combination fights itself.
If your service is stuck on "tool goes slow, the whole queue jams," this paper is worth reading. If most of your load is chat with no long tool chain, the payoff may be far below the number in the abstract.
As for what agent-service scheduling should actually optimize — there is not even a consensus objective function yet: TTFT, end-to-end latency, or throughput? TAO takes a lexicographic triplet as its tradeoff, and the price is that negative 504% TTFT regression on AutomationBench. The next thing worth watching may not be a smarter scheduler, but first agreeing on what we are optimizing.