[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tao-tool-aware-agentic-serving-scheduler":3,"topics-all":38,"news-related-8055f566-5501-4ccf-ae9d-22379d4d3da7":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"8055f566-5501-4ccf-ae9d-22379d4d3da7","TAO调度:工具等待也算进Agent的调度收益","Agent 服务在 GPU 推理与 CPU 工具之间来回切换，工具要等多久一直不进调度账。上海交大与 Stanford 的 TAO 把工具返回就绪信号接进估值三元组，用动态规划联合决定准入与放置。作者自报延迟最多降 26.8%；但 12 组配置中位数只有 4.6%，36 个格子里 5 个是负的。","一个 Agent 会话跑在集群上时，GPU 并不是一直在算。更多时候它在等——等编译、等检索、等一条命令跑完。这段\"等\"的时间，恰恰是决定整套系统快慢的关键变量。\n\n上海交大与 Stanford 的一项工作重算了这笔账。论文 [TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.37230)（arXiv:2610.37230，10 月 27 日提交）把\"工具什么时候返回\"当成和 KV cache 命中、token 占用并列的第三个调度信号，再用动态规划同时决定\"接下哪些会话、放到哪个节点\"。作者自报的评测里，最高能拿到延迟降 26.8%、TTFT 降 36.3%、解码吞吐涨 29.5%。\n\n## 现有调度漏掉的那笔账\n\n论文把这个场景叫 CGDC：GPU 聚成推理集群，工具执行散落在 CPU 上，一个会话在两者之间横跳。这意味会话对 GPU 的需求是**可打断**的——工具在跑时它不占 GPU。\n\n现有系统普遍是\"先评估、再决策\"两段式：评估看 GPU 侧特征（KV 前缀能不能复用、要烧多少 token），决策决定准入和放置。可这套评估里根本没有\"工具什么时候回来\"这一项。看不见返回时机，等于在给一份残缺的估值排序。\n\nTAO 给每个\"会话 × 节点\"配一个三元组估值：**随时间衰减的 cache credit**、**二元就绪信号**、**token 占用**，按字典序比较。cache credit 用 2 的负指数衰减，半衰期 10 秒——可复用 token 越久远，打折越狠。就绪信号来自工具时长预测：优先用直接计时证据，其次退回工具自身或全局历史，没有就用 5 秒默认值。选谁、放哪，用容量状态动态规划一次解完，应用前重新校验。\n\n## 最亮的一格和最暗的一格\n\n作者自报的数字来自 12 组\"模型 × 负载 × 并发\"配置。把表 1 十二组延迟改善排一遍，中位数只有 4.6%，6 组不超过 5%；26.8%、36.3%、29.5% 三个峰值全来自同一格——DeepSeek-V4 Flash 在 SWE-bench Lite 上、64 workers。\n\n最暗的一格更值得看。表 3 是混合负载与 chatbot 场景的兼容性测试：Terminal-Bench + SWE-Lite 混跑 389 个任务，三档并发下延迟分别改善 12.2%、23.2%、13.8%；而 AutomationBench 800 任务那行、128 workers 一档，延迟是负的 7.0%，TTFT 是负的 504.0%（从 Dynamo 的 3.166 秒涨到 TAO 的 19.123 秒），解码吞吐负 11.2%。整张表 36 个格子里，5 个是负的。\n\n这不一定是 bug。工具就绪信号对纯聊天负载本来就没意义——没有工具要等，它就是噪声，甚至会把调度带偏。论文把它放在兼容性章节而非摘要，算是诚实。但\"延迟最多降 26.8%\"要读成\"最好情况下\"，不是\"平均\"。\n\n## 拆开看，哪块组件在出力\n\n表 4 是消融，跑在 DeepSeek-V4 Flash、SWE-Lite、32 workers 上，完整配置解码吞吐 1417.97 tok\u002Fs，前缀缓存命中 96.31%，TTFT 均值 0.154 秒。逐个拆：\n\n- 去掉 cache credit：1398.35 tok\u002Fs\n- 去掉就绪信号：1363.06 tok\u002Fs\n- 去掉 session evaluation：1355.20 tok\u002Fs\n- 去掉 decision making（DP 换成 best-fit-decreasing）：1233.72 tok\u002Fs\n\n掉得最狠的是最后一项——决策端比评估端更值钱。难处在容量约束下同时决定\"选谁\"和\"放哪\"，而这恰是启发式方法最容易退化的地方。\n\n第 4.3 节拆得更细：固定 218 个任务的负载回放里，cache token 排序让前缀命中率比 ThunderAgent 高 0.19 个百分点、平均暂停等待降 4.5%；叠加 cache credit 衰减后，命中率再高 0.31 个百分点、解码吞吐涨 1.68%。开启就绪信号，解码吞吐涨 4.03%、工具返回到首 token 延迟降 10.88%，平均暂停等待从 1.067 秒降到 0.972 秒。\n\n## 预测准不准，其实不是重点\n\n这是论文最被低估的一点。附录 G 做了个\"神谕\"实验：把回放里已知的工具真实时长喂给预测器。预测误差降低 99.53%–99.69%，降到 0.025–0.030 秒。\n\n但收益呢？makespan 改善在 −0.10% 到 +2.24% 之间，平均会话延迟在 −0.36% 到 +0.71% 之间。基本等于没。\n\n真正有影响的是另一个数：工具延迟越长，就绪信号影响到的 DP 放置决策越多。scale 1（每次工具返回额外加 1–5 秒）时是 668 次决策里的 11 次、占 1.65%；scale 2 是 1345 次里的 53 次、3.94%；scale 4（加 4–20 秒）是 2521 次里的 200 次、7.93%。工具越慢，\"先别占 GPU、留给马上要回来的会话\"越值钱。所以 TAO 的收益不是\"预测得更准\"，而是\"把一个被忽略的维度接进了决策结构\"。\n\n代价在附录 I：四副本 Qwen 在 96–160 workers 下规划提交从 98 次涨到 1681 次，160 workers 时分配求解平均 1.59 毫秒、P95 3.68 毫秒，两部分合计 4.03 秒、占整轮 0.112%。DP 搜索也有兜底：64 workers 的 DeepSeek 跑里有 5.24% 的提交走了回退，Qwen 那组在 3.06%–8.98% 之间。\n\n## 我的判断\n\n方向是对的。上一代 LLM 推理服务优化的是\"把一个请求算快\"，这一代要优化的是\"在等待和计算的交替里把整队排好\"。工具时间不进调度估值，等于把 Agent 服务当成不断线的 LLM 服务在运营——这是不少自建 Agent 平台的真实盲区。\n\n但别把论文的数字当承诺。同一批配置的中位数看，延迟收益 4.6%、TTFT 中位数 14.75%、解码吞吐中位数 4.35%；只有\"负载高度重复 + 并发顶到某个区间\"时才跳到 26.8%。失效区也明确：纯聊天负载、高并发下 TTFT 甚至可能差一个数量级，单节点低并发那档也没赢过 vLLM。换个模型看是另一番景象——GLM-4.7 跑 SWE-verified，延迟降 16.2%–18.9%、TTFT 降 15.7%–38.4%、解码吞吐涨 22.8%–24.7%。\n\n还有个信号在附录 E.2：和 TokenCake 这类引擎层 KV 管理器组合时，\"只加 cache-aware\"反而比单用 TokenCake 更差（16 workers 下延迟 137.907 秒 vs 126.891 秒），而\"TAO + TokenCake\"是 122.940 秒。调度层和引擎层的缓存管理不是简单叠加，选错组合会互相打架。\n\n如果你的服务正卡在\"工具一慢、整队就堵\"上，这篇值得读；如果负载大部分是聊天、没有长工具链，收益远小于摘要里那个数字。\n\n至于\"Agent 服务的调度到底该优化什么\"——现在连一个公认的目标函数都没有：TTFT、端到端延迟、还是吞吐？TAO 用字典序三元组做取舍，代价就是 AutomationBench 上那 504% 的 TTFT 负收益。下一个方向也许不是更聪明的调度器，而是先把\"我们到底要优化什么\"说清楚。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.37230","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"fca9258a-9430-455a-b95d-b9fae5e373a8","ai-inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"4b6144ff-33a3-43fc-8508-2c79521382b2","en","TAO: Tool-Return Readiness as a Scheduling Signal","TAO adds tool-return readiness to agent serving, then places sessions by dynamic programming. Authors report up to 26.8% lower latency, median 4.6%.","When an agent session runs on a cluster, the GPU is not computing most of the time. More often it is waiting — waiting for a compiler, a retriever, or a shell command to finish. That waiting time is exactly the variable that decides how fast the whole system runs.\n\nA team from Shanghai Jiao Tong University and Stanford took another run at that accounting. The paper [TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.37230) (arXiv:2610.37230, submitted October 27) treats \"when will the tool come back\" as a third scheduling signal, on a par with KV cache hits and token footprint, and then uses dynamic programming to jointly decide which sessions to admit and which node to place them on. In the authors' own evaluation, one configuration — DeepSeek-V4 Flash on SWE-bench Lite at 64 workers — shows latency down 26.8%, mean time to first token (TTFT) down 36.3%, and decode throughput up 29.5%.\n\n## The bill existing schedulers miss\n\nThe paper calls this scenario CGDC: GPUs clustered for inference, tool execution scattered across CPUs, with a single session hopping back and forth between the two. That makes a session's demand for GPU capacity *interruptible* — while a tool is running, the session is not holding the GPU.\n\nExisting systems mostly follow an evaluate-then-decide pattern. Evaluation looks at GPU-side session features (whether the KV prefix can be reused, how many tokens will be burned), and a later stage decides admission and placement. The problem is that \"when will the tool come back\" is simply not part of that evaluation. A scheduler that cannot see tool-return timing is effectively ranking sessions against an incomplete valuation.\n\nTAO's fix is to assign each session-node pair a triplet valuation: **cache credit with time-aware decay**, a **binary readiness signal**, and **token count**, compared lexicographically. Cache credit decays as a negative power of two with a 10-second half-life — the longer ago a reusable token was observed, the harder it is discounted. The readiness signal comes from a tool-duration predictor that prefers direct timing evidence, falls back to per-tool history or global history, and finally to a 5-second default. Which sessions to take and where to put them are solved in one shot by a capacity-state dynamic program, which revalidates against live state before applying.\n\n## The brightest cell and the darkest one\n\nThe headline numbers come from 12 model-workload-concurrency configurations. Sort the twelve latency improvements in Table 1 and the median is only 4.6%, with six of them at or below 5%. The 26.8% cell comes from DeepSeek-V4 Flash on SWE-bench Lite at 64 workers.\n\nThe darkest cell is more interesting. Table 3 tests workload compatibility on mixed and chatbot scenarios. Terminal-Bench mixed with SWE-Lite over 389 tasks shows TAO improving latency by 12.2%, 23.2% and 13.8% across the three concurrency levels. But the AutomationBench row of 800 tasks at 128 workers goes the other way: latency is negative 7.0%, TTFT is negative 504.0% (from Dynamo's 3.166 seconds up to TAO's 19.123 seconds), and decode throughput is negative 11.2%. Across the whole table, 5 of 36 cells are negative.\n\nThat is not necessarily a bug. A tool-readiness signal is meaningless for a pure chat workload — with no tool to wait for, it is noise, and it may actively pull scheduling off course. The paper puts this in the compatibility section rather than the abstract, which is fair. But the correct reading of \"latency reduced by up to 26.8%\" is \"best case\", not \"typical case\".\n\n## Which component is doing the work\n\nTable 4 is the ablation, run on DeepSeek-V4 Flash, SWE-Lite, 32 workers. The full configuration decodes at 1417.97 tok\u002Fs with 96.31% prefix-cache hit rate and 0.154 seconds mean TTFT. Removing pieces one at a time:\n\n- without cache credit: 1398.35 tok\u002Fs\n- without readiness: 1363.06 tok\u002Fs\n- without session evaluation: 1355.20 tok\u002Fs\n- without decision making (DP replaced by best-fit-decreasing): 1233.72 tok\u002Fs\n\nThe last one hurts most — the decision side is worth more than the evaluation side. The hard part is deciding *what* to take and *where* to put it simultaneously under capacity constraints, and that is exactly where heuristic methods tend to degrade.\n\nSection 4.3 breaks it down further. In a fixed replay of 218 tasks, cache-token ordering raises the prefix-cache hit rate by 0.19 percentage points over ThunderAgent and cuts mean pause wait by 4.5%; adding cache-credit decay lifts the hit rate by another 0.31 points and decode throughput by 1.68%. Turning on the readiness signal adds 4.03% decode throughput and cuts tool-return-to-first-token delay by 10.88%, with mean pause wait falling from 1.067 to 0.972 seconds.\n\n## Prediction accuracy is not the point\n\nThis is the part of the paper I think is most underrated. Appendix G runs an oracle experiment: feed the predictor the true tool duration from the replay. Prediction error drops by 99.53%–99.69%, down to 0.025–0.030 seconds.\n\nBut the payoff? Makespan improvement lands between −0.10% and +2.24%, and mean session latency between −0.36% and +0.71%. Essentially nothing.\n\nWhat does matter is a different number: the longer the tool delay, the more DP placement decisions the readiness signal touches. At scale 1 (an extra 1–5 seconds per tool return) it changes 11 of 668 decisions, or 1.65%; at scale 2, 53 of 1345, or 3.94%; at scale 4 (an extra 4–20 seconds), 200 of 2521, or 7.93%. The slower the tool, the more valuable the judgment \"don't grab the GPU yet, save it for the session about to come back.\" So TAO's gain does not come from predicting better — it comes from wiring an ignored dimension into the decision structure.\n\nThe cost is in Appendix I: across four-replica Qwen runs at 96–160 workers, planning submissions rise from 98 to 1,681, and at 160 workers a completed assignment solve averages 1.59 ms with a P95 of 3.68 ms, the two measured parts together accounting for 4.03 seconds, or 0.112% of the run. The DP search also has a fallback: 5.24% of submissions at 64 DeepSeek workers fell back to best-fit-decreasing, and the Qwen sweep ranged from 3.06% to 8.98%.\n\n## My read\n\nThe direction is right. The difference between last-generation LLM inference serving and agent infrastructure is precisely this: the old problem was \"make one request fast,\" the new one is \"keep the whole queue ordered while computation and waiting alternate.\" Leaving tool time out of the valuation means running an agent service as if it were an LLM service that never pauses — and that is a real blind spot for plenty of self-hosted agent platforms today.\n\nBut do not treat the paper's numbers as a promise. At the median of the same configuration set, latency gains are 4.6%, TTFT median 14.75%, decode throughput median 4.35%; only when the load is highly repetitive and concurrency tops a particular band do you reach 26.8%. The failure region is clear too: pure chat workloads, where at high concurrency TTFT can be an order of magnitude worse, and the single-node low-concurrency band, where TAO did not beat vLLM either. A different model tells a different story — on GLM-4.7 with SWE-verified, the paper reports latency down 16.2%–18.9%, TTFT down 15.7%–38.4%, and decode throughput up 22.8%–24.7%.\n\nOne more signal hides in Appendix E.2: when paired with an engine-level KV manager like TokenCake, adding cache-awareness alone is actually *worse* than using TokenCake by itself (at 16 workers, 137.907 seconds of latency versus 126.891), while TAO + TokenCake gives 122.940. Scheduling-layer judgment and engine-layer cache management do not simply compose; the wrong combination fights itself.\n\nIf your service is stuck on \"tool goes slow, the whole queue jams,\" this paper is worth reading. If most of your load is chat with no long tool chain, the payoff may be far below the number in the abstract.\n\nAs for what agent-service scheduling should actually optimize — there is not even a consensus objective function yet: TTFT, end-to-end latency, or throughput? TAO takes a lexicographic triplet as its tradeoff, and the price is that negative 504% TTFT regression on AutomationBench. The next thing worth watching may not be a smarter scheduler, but first agreeing on what we are optimizing.\n","tao-tool-aware-agentic-serving-scheduler","2026-10-29T11:16:02Z","2026-10-29T11:17:36.872345Z","2026-10-29T11:17:36.872364Z",true,"agent",258,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"1ec34a26-f012-4bc4-8193-14e651c07516","程序进化搜出记忆设计:涨12个百分点,成本只占35%","mape-agent-memory-program-evolution","2026-11-14T05:05:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"417ea170-637f-48be-a998-aa13ec5cdb10","记忆税:11个记忆系统多数跑输朴素RAG","memory-tax-agent-memory-net-value","2026-10-28T21:07:37+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"f2f2f891-a350-4bd1-9055-6eea0c72de66","Agent 的 KV 缓存不该由系统单方面接管：清华团队把缓存控制权交回 Agent 侧，并给出 AKV 推理引擎","akv-cache-engineering-agents-kv","2026-10-22T07:09:11+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"00346b75-f071-42fd-ae16-db4c5569f01a","EarlyEval 提前叫停注定失败的 Agent:近半 token 省下,分辨率只动一两个点","earlyeval-early-stop-agent-eval","2026-09-03T21:04:52+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"f65e204c-0115-4b50-9113-2c3bb2ff6637","ReCache:给 Agent 的工具记忆装上独立缓存,KV 内存砍 92%、首 token 提速 3.655 倍","recache-agent-kv-cache-reuse","2026-08-24T15:30:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"822987d9-faaa-492d-b7ba-1c0427f72f9b","AgentOps 出海第一步：拆腾讯云 ADP 4.0 海外版的「Agent+Workflow 分账」架构","tencent-adp-4-0-overseas-agentops","2026-07-20T02:11:09+00:00"]