[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-thunderagent-agentic-inference-icml26-spotlight-2602-13692":3,"topics-all":36,"news-related-04c87475-dc8b-401f-b078-599bc6f71e55":55},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":34,"view_count":35},"04c87475-dc8b-401f-b078-599bc6f71e55","ThunderAgent：把每次 Agent 任务当成\"程序\"调度，把 KV cache 抖动打掉，推理吞吐翻倍","Together AI 联合 Georgia Tech、UIUC、CMU 把 Agentic 推理的调度粒度从「请求」上提到「程序」，在 8×H100 节点上把合成数据生成吞吐做到 SGLang 同条件下的两倍以上，论文中稿 ICML 2026 Spotlight。","# ThunderAgent：把每次 Agent 任务当成\"程序\"调度，把 KV cache 抖动打掉，推理吞吐翻倍\n\n2026 年 7 月 29 日，Together AI 工程团队联合 Georgia Tech、UIUC、CMU 在官方博客发布 Agentic 推理调度框架 **ThunderAgent**，并同步开源代码与论文（arXiv:2602.13692）。这篇已被 ICML 2026 接收为 **Spotlight 论文**的工作，直接瞄准当下推理框架在大规模 Agent 并发场景下的一个老毛病：**KV cache 抖动（KV cache thrashing）**。\n\n## 背景：现在的 vLLM \u002F SGLang \u002F TensorRT-LLM 为什么在 Agent 场景\"水土不服\"\n\n文章把问题描述得相当直白——这些主流推理引擎都是「**请求级（request-level）**」调度器，每一个独立的 LLM 调用对调度器来说就是一个原子单位，看不见这是一个长链路 Agent 任务的一部分。\n\n典型的 Agent 工作流是这样的：\n\n- **reasoning 阶段**：模型在 GPU 上疯狂吐 token；\n- **acting 阶段**：Agent 在等外部工具返回（比如编译器、搜索、shell），GPU 完全空转。\n\n一个 8 卡 H100 节点同时跑上百个 Agent，每一轮的 KV cache 都在涨。当节点显存吃紧的时候，老调度器机械地按 **LRU** 踢 cache——Agent A 刚停下来等一个工具调用，它的 KV cache 就被踢掉给 Agent B 让路；几秒后 Agent A 的工具返回，调度器只能**从头重新预填**它的整段对话历史，又顺手把 Agent C 的 cache 踢掉。这样连锁反应下去，就是 ThunderAgent 命名里的 **thrashing（抖动）**。\n\n文章的 baseline 数据很扎心：同样 8×H100 + HiCache offload、batch size 192，SGLang 默认调度器吞吐只跑出 **390 token\u002Fs、平均延迟 65 秒**。\n\n## ThunderAgent 的解法：把整个工作流抽象成\"可调度的程序\"\n\nThunderAgent 不是又写了一个推理引擎，而是在**调度粒度**上做文章。它是一个轻量级调度层，夹在 Agent 客户端和推理后端之间，引入了一个新概念 ——**「程序级（program-level）」调度**。\n\n具体三板斧：\n\n1. **把 Agent 工作流抽象成 program**：调度器维护一张 program table，记录每一个 Agent 程序当前在哪一阶段（reasoning 还是 acting）、占用多少 KV cache、落在哪个节点上。\n2. **主动暂停低优先级 program**：显存吃紧时不是粗暴踢 cache，而是**暂停**整个低优先级的工作流，让活跃工作的 KV cache 命中率稳住。\n3. **全局等待队列 + 多节点均衡**：被暂停的 program 恢复时，由调度器送到**当时空余 KV 容量最大的节点**去，而不是死绑在原来的节点。这解决了 SGLang Gateway 那种静态 pin 节点策略在长尾上下文下的内存失衡问题。\n\n另外，ThunderAgent 跟 HiCache \u002F LMCache 这些 KV cache offloading 方案是**叠加关系**，不是替代。它把 GPU HBM、CPU RAM、磁盘三层 cache 看成\"统一池\"，offload 不再是缓兵之计，能跟 program-level 调度配合真正\"治本\"。\n\n## 数字说话\n\nTogether AI 在自家 **CoderForge** 合成数据生成流水线（上百个 Agent 跑沙箱、几十轮 code trajectory）上做的内部对比：\n\n- **单节点**：batch 192 时，吞吐从 SGLang 的 390 token\u002Fs 提到 ThunderAgent 的 **803 token\u002Fs**（≈ 2.06×），平均延迟从 65 秒降到 **10.6 秒**（≈ 6×）。\n- **8 节点 64 卡**：吞吐从 16 卡 671 steps\u002Fmin 线性扩到 **2,248 steps\u002Fmin**，相对 SGLang Gateway 的领先优势从 2 节点的 1.79× 拉到 8 节点的 **2.39×**。\n- **接入成本极低**：论文强调 ThunderAgent 对客户端**只需要加一个 `program_id` 字段**，用 OpenAI-compatible 接口，对现有 speculative decoding、量化这些优化完全 pass-through。\n\n文章还提到 ThunderAgent 已经**被 SkyRL 和 NVIDIA Dynamo 接入**——Dynamo 这种 NVIDIA 刚发力的分布式推理框架愿意接，说明 program-level 这个抽象在工业界也会被验证。\n\n## 所以呢：调度层的\"下一个 Transformer 级\"机会？\n\n推理框架的演进过去两年基本是**「让单卡跑得更快」**这条线（PagedAttention、continuous batching、prefix caching、FlashAttention）；进入 Agent 时代以后，工作负载从「一次性问答」变成**「几十轮带工具的长程任务」**，调度器的视野必须从「请求」上提到「程序」。\n\nThunderAgent 的核心 insight 其实只有一句话：**scheduler 一旦能看见\"两个相邻 LLM 调用其实属于同一个 Agent\"，它就能比 LRU 做更聪明的取舍**。这套抽象和 Thompson Sampling、Speculative Decoding 一样属于\"看起来朴素，落地就跑出来\"的工程 trick——但作者来自 Georgia Tech\u002FUIUC\u002FCMU\u002FTogether AI 这套组合，决定了它很快会被主流框架吸收。\n\n对国内搞 Agent Infra 的同学来说，下一个值得盯的也许不是再写一个 inference engine，而是**接住 program-level 调度这个抽象**——尤其是在 long-horizon 多 Agent 协作或国产 GPU 多卡环境下的中文工程实现，目前公开实现还很稀疏。\n\n> 参考资料：[Together AI 官方博客](https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fthunderagent)、[arXiv 论文 2602.13692](https:\u002F\u002Farxiv.org\u002Fabs\u002F2602.13692)、[GitHub 开源仓库](https:\u002F\u002Fgithub.com\u002FThunderAgent-org\u002FThunderAgent)","https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fthunderagent","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"fca9258a-9430-455a-b95d-b9fae5e373a8","ai-inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"d290ad7e-c5df-499e-be9a-c2fe56fe30f0","en","ThunderAgent schedules agent runs as programs, doubles throughput","Together AI, jointly with Georgia Tech, UIUC, and CMU, raises agentic-inference scheduling granularity from \"request\" to \"program\". On 8×H100, their internal synthetic-data-generation pipeline more than doubles throughput and cuts latency ~6× versus SGLang under identical settings. The paper is an ICML 2026 Spotlight.","# ThunderAgent: Scheduling Agentic LLM Inference at Program Granularity, Killing KV Cache Thrashing and Doubling Throughput\n\nOn July 29, 2026, the Together AI engineering team, in collaboration with Georgia Tech, UIUC, and CMU, announced **ThunderAgent** — a new scheduling framework purpose-built for agentic LLM inference. Code and paper (arXiv:2602.13692) were released in lockstep. The work has been accepted as an **ICML 2026 Spotlight** and squarely attacks the dirty secret of every mainstream inference engine running large-scale agent workloads today: **KV cache thrashing**.\n\n## Background: Why today's vLLM \u002F SGLang \u002F TensorRT-LLM engines over-promise under agent loads\n\nThe blog post frames the diagnosis in plain English. Every one of these engines is a **request-level scheduler**: every individual LLM call is treated as an atomic unit, with no visibility into the fact that it belongs to a longer-running agentic workflow.\n\nA typical agentic workload oscillates between two phases:\n\n- a **reasoning** phase, where the GPU is busy churning out tokens;\n- an **acting** phase, where the model blocks on an external tool (compiler, search, shell, etc.) and the GPU is completely idle.\n\nRun hundreds of such agents concurrently on an 8×H100 node and every agent's KV cache grows during each phase, competing for the same finite pool of GPU memory. Under memory pressure, the legacy scheduler panics and falls back to the obvious eviction policy: LRU. Agent A pauses waiting for a tool call, its KV cache gets evicted to make room for Agent B's prefill. Seconds later Agent A's tool returns, and the engine has to **re-prefill Agent A's entire conversation history from scratch**, which in turn boots Agent C's cache out. The cascade is what the authors call **thrashing** — and it is exactly what ThunderAgent takes its name from.\n\nThe baseline numbers in the paper are sobering. On 8×H100 with HiCache offloading and a batch size of 192, SGLang's default scheduler tops out at **390 token\u002Fs with a mean latency of 65 seconds**.\n\n## How ThunderAgent fixes it: turning every workflow into a schedulable program\n\nThunderAgent is not a new inference engine. The novelty sits one layer above: in the **granularity of scheduling decisions**. It is a lightweight layer that lives between agentic clients and inference backends, and the primitive it schedules on is the **program-level workflow**, not the individual request.\n\nThree concrete moves:\n\n1. **Workflow abstraction as a program.** The scheduler keeps a program table that tracks every agent workflow's current phase (reasoning or acting), its KV-cache footprint, and which node it currently lives on.\n2. **Active pausing of low-priority programs.** Under memory pressure, instead of blindly evicting cache, the scheduler **pauses** low-priority workflows entirely so the remaining active ones keep their KV cache hot.\n3. **Global waiting queue + load balancing.** Resumed programs are routed to whichever node currently has the most spare KV capacity, rather than being statically pinned. This avoids the memory imbalance that SGLang Gateway's session-based static pinning creates when agent context lengths get long-tailed.\n\nCrucially, ThunderAgent **composes** with KV-cache offloading schemes like HiCache and LMCache rather than replacing them. It treats GPU HBM, CPU RAM, and disk as one unified cache pool, so offloading stops being a temporary patch and starts working in earnest alongside program-level scheduling.\n\n## What the numbers look like\n\nTogether AI ran its own internal **CoderForge** synthetic-data-generation pipeline (hundreds of coding agents in sandboxes, dozens of turns each) as the testbed:\n\n- **Single node, batch 192:** throughput goes from SGLang's 390 token\u002Fs to ThunderAgent's **803 token\u002Fs** (≈ 2.06×), and mean latency drops from 65 s to **10.6 s** (≈ 6×).\n- **8 nodes \u002F 64 GPUs:** throughput scales near-linearly from 671 steps\u002Fmin at 16 GPUs to **2,248 steps\u002Fmin**. The speedup over SGLang Gateway widens from **1.79× at 2 nodes to 2.39× at 8 nodes**.\n- **Drop-in ergonomics:** the only client-side change is adding a `program_id` field. The OpenAI-compatible interface and pass-through compatibility with existing optimizations (speculative decoding, quantization) are preserved.\n\nThe blog also notes that ThunderAgent has already been **integrated into SkyRL and NVIDIA Dynamo** — and Dynamo in particular is NVIDIA's big bet on disaggregated inference. The fact that it is willing to plug in a third-party scheduler at this layer suggests the program abstraction will land in production framework roadmaps sooner than later.\n\n## So what: is \"the next Transformer moment\" hiding in the scheduler?\n\nFor the past two years, inference-stack improvements have mostly been about **\"make one request go faster\"** — PagedAttention, continuous batching, prefix caching, FlashAttention. The agent era changes the shape of the workload from \"one-shot Q&A\" to **\"long, multi-turn, tool-using workflows\"**, and the scheduler's window of visibility has to move from request up to program.\n\nThe single sentence that captures ThunderAgent is: **once the scheduler can see that two adjacent LLM calls belong to the same agent, it can make smarter trade-offs than LRU ever will.** Like Thompson Sampling or Speculative Decoding, it looks trivially obvious in hindsight — but the author affiliation stack (Georgia Tech \u002F UIUC \u002F CMU \u002F Together AI) suggests that mainstream inference frameworks will absorb the abstraction quickly.\n\nFor builders of agent infrastructure, the immediate follow-up is not \"write yet another inference engine\" but **\"be the first to ship a clean, well-engineered implementation of program-level scheduling for multi-agent, long-horizon workloads\"** — especially in environments with non-NVIDIA accelerators and Chinese-language tooling, where public implementations are still sparse.\n\n> References: [Together AI blog post](https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fthunderagent) • [arXiv paper 2602.13692](https:\u002F\u002Farxiv.org\u002Fabs\u002F2602.13692) • [GitHub repo](https:\u002F\u002Fgithub.com\u002FThunderAgent-org\u002FThunderAgent)","thunderagent-agentic-inference-icml26-spotlight-2602-13692","2026-08-04T00:00:00Z","2026-08-03T16:13:23.665889Z","2026-08-03T16:13:23.665897Z",true,"agent","https:\u002F\u002Fcdn.prod.website-files.com\u002F69654e88dce9154b5f12070c\u002F6a6a82035816b6fcc9af706_20260729_ThunderAgent_1200x630.jpg",297,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":32,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":32,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"0565190a-0bcd-492f-934f-0ad2ab32f485","70万参数2.8MB填一张表:Cua开源CUA-S1,单次前向替代23轮LLM","cua-s1-forms-system-one-model","2026-09-20T13:11:48+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"13378d5e-2440-496d-8c3c-7d36858e641d","不聊天的端侧基座:Needle 3 用 8-29MB 在微控制器上跑工具调用","needle-3-tiny-tool-calling-model","2026-09-19T13:09:46+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"c696208b-6535-4eb9-b1ed-2e4f835d2f88","NVIDIA SoL-Pi 把 coding agent 的 token 砍掉 44%,harness 开始变天","nvidia-sol-pi-harness-token-compression","2026-09-19T03:00:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"9ba1770e-87f2-47b9-aaa1-19f4f2ba78f1","ScienceIDE:把全球科学代码变成智能体训练场","scienceide-scientific-code-agent-environments","2026-09-17T23:05:17+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"0fe869ca-11c4-4831-afb1-19a31fd88dfc","智谱公开国内大模型首个 RSI:GLM-5.3 Infra Agent 在 10 万国产卡集群自建推理,2 周吞吐 3 倍","zhipu-glm-rsi-infrastructure-chinese-cluster","2026-09-17T08:00:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"cb1e799d-d6d7-4ab9-9eaf-bea0aa432b06","Mistral 模型进驻 Firefox:119B 开放权重模型驱动浏览器 AI 助手","mistral-small-4-firefox-smart-window","2026-09-16T17:07:00+00:00"]