[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-toktier-stateful-tokenization-agent-serving":3,"news-related-4bbc55d2-cabc-477f-a3ad-4e2c119aff2a":39},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":37,"view_count":38},"4bbc55d2-cabc-477f-a3ad-4e2c119aff2a","TokTier 抓住 Agent 推理的隐藏瓶颈：缓存命中 94.1%，分词仍吃掉 64% 首 token 时间","TokTier 针对编码 Agent 反复提交“超长历史 + 短追加”的流量，把 tokenizer 从无状态全量重算改为精确的有状态增量修复。论文称其在 17 类 tokenizer 上保持 token ID 零差异，并将 vLLM 首 token 中位延迟降低 16%–34%，揭示高缓存命中率之后的新推理瓶颈。","# TokTier：Agent 推理的新瓶颈，竟然是分词器\n\n当提示词缓存命中率已经做到 **94.1%**，为什么 Agent 的首 token 还是慢？一篇 7 月 31 日提交的论文给出了一个反直觉答案：GPU 可能没在算模型，而是在等 CPU 把几百万字符重新分词。\n\n## KV Cache 省下来了，前端却还在重复劳动\n\n普通聊天通常一次发来一段新文本；编码 Agent 的流量完全不同。它会把很长的历史轨迹保留在请求里，每执行一次工具，只在末尾追加一小段结果，然后再次提交整段上下文。\n\nTokTier 团队分析了两个 Agent 生态中的 **153,951 次调用**：一次续写的中位新增量约为 1,400 个字符，只有 1.0%–3.6% 的调用会新建或重建会话，但完整上下文最长可达数百万字符。后端能复用 KV Cache，很多前端分词器却仍从头扫描全文。随着提示词缓存命中率接近 99%，论文的组件测量显示，分词过程可从首 token 时间的 10% 膨胀到 **64%**。\n\n这不是把 tokenizer 换成更快的 Rust 实现就能彻底解决的问题。Agent 请求的真正特征是“巨大旧前缀 + 很短新尾巴”，系统需要复用上一次的分词状态，而不是每轮把相同前缀重做一遍。\n\n## 只重算边界，但结果必须逐 token 一致\n\n难点在于，BPE 分词并非简单追加。新字符可能改变旧文本末尾的 token 边界，粗暴拼接会让 token ID 与完整分词结果不一致，进而破坏 KV Cache 对齐。\n\nTokTier 的做法是保存会话上一轮的 token 序列，只在新增文本附近重新分词，并执行逐请求的“稳定边界”检查。检查通过才拼接；不通过就扩大窗口，仍不安全则退回完整分词。它追求的不是近似，而是一个明确契约：**输出 token ID 必须与参考分词器从头处理全文完全相同**。对无法复用前缀的新请求，系统再把 GPT 系列 tokenizer 的预分词与 BPE 搬到 GPU 上执行，并用抽样影子校验监控线上偏差。\n\n论文在 17 个 tokenizer 家族上做了 150 亿次切分检查，覆盖 12.4 TB 真实文本和超过 9.3 万个 Agent 步骤，报告零差异。对于 10 万到 300 万字符的上下文，增量修复耗时为 **0.5–1.1 毫秒**，相对 Hugging Face tokenizer 最多快 437 倍；GPU 完整分词处理 100 万字符只需 0.87 毫秒。\n\n## 真正重要的，是端到端延迟\n\n接入 vLLM 后，TokTier 把首 token 中位延迟降低 **16%–34%**，突发流量下 P99 降低 23%。在 50 毫秒 P99 目标下，4 个修复 CPU 核加 1 张 GPU 可支撑 1,821 请求\u002F秒，而 16 核无状态前端在 40 请求\u002F秒就饱和。\n\n但这些数字仍需谨慎解读：论文目前是预印本，主要基于两个 Agent 生态和特定硬件配置；引入有状态服务还会带来会话路由、状态一致性、GPU 利用率及故障恢复成本。它并不意味着每个聊天接口都该加一层 GPU tokenizer。\n\n更值得关注的是系统优化逻辑变了。过去大家盯着模型计算、KV Cache 和投机解码；当这些环节被逐步加速，原本不起眼的 CPU 前处理会成为新的最长板。**Agent 时代的推理优化，不再只是让模型算得快，而是让整条请求链少做重复工作。**","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.29678","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"24c2843e-e036-4176-a465-8cdf50a94e1a","en","TokTier: tokenization eats 64% of first-token time","TokTier targets the traffic pattern common in coding agents: a very long historical context followed by a small appended tool result. Instead of re-tokenizing the full request on every call, it performs exact stateful incremental repair around the append boundary, with a fallback to full tokenization whenever the token IDs could diverge. The paper reports zero divergence across 17 tokenizer families and says vLLM integration reduced median time to first token by 16%–34%. Its broader lesson is that once KV-cache reuse is highly effective, frontend tokenization can become a major serving bottleneck.","# TokTier: The Hidden Bottleneck in Agent Inference Is Tokenization\n\nWhen prompt-cache hit rates have already reached **94.1%**, why can an agent still take a long time to produce its first token? A paper submitted on July 31 offers a counterintuitive answer: the GPU may not be waiting on model computation at all. It may be waiting for the CPU to tokenize millions of characters again.\n\n## KV cache reuse does not remove repeated frontend work\n\nA normal chat request usually contains a new piece of text. Coding agents generate a very different traffic pattern. They retain a long history of actions and observations, append a small tool result after each step, and submit the entire context again.\n\nThe TokTier authors analyzed **153,951 calls** from two agent ecosystems. The median continuation appended roughly 1,400 characters. Only 1.0%–3.6% of calls started or rebuilt a session, yet the complete contexts in those calls could reach millions of characters. The serving backend may reuse the KV cache, while the frontend tokenizer still scans the full request from scratch. In the paper's component measurements, as the prompt-cache hit rate approaches 99%, tokenization grows from about 10% to **64% of time to first token**.\n\nSimply replacing a tokenizer with a faster Rust implementation does not solve the entire problem. The defining pattern is “a huge old prefix plus a short new suffix.” The serving stack needs to reuse the previous tokenization state instead of repeating work on the same prefix at every turn.\n\n## Recompute the boundary, but preserve exact token IDs\n\nThe difficult part is that BPE tokenization is not a simple append operation. New characters can change token boundaries near the end of the previous text. If a system concatenates token sequences naively, its token IDs may differ from a full reference tokenization, breaking the alignment required for KV-cache reuse.\n\nTokTier stores the previous token sequence for a session and re-tokenizes only a window around the newly appended text. It then performs a per-request stable-boundary check. If the boundary is safe, the new result is spliced into the existing sequence. If not, the window is widened; if the result is still unsafe, the system falls back to full tokenization.\n\nThe goal is not an approximation. TokTier makes a strict promise: **the emitted token IDs must be identical to those produced by the reference tokenizer when it processes the complete request from the beginning**. For requests without a reusable prefix, the system decomposes GPT-family regular-expression pre-tokenization into run-local rules and runs exact pre-tokenization and BPE on a GPU. A sampled shadow verifier checks live traffic for divergence.\n\nThe paper reports 15 billion split checks across 17 tokenizer families, a 12.4-terabyte real-text corpus, and more than 93,000 replayed agent steps, with zero divergence in those campaigns. Incremental repair takes **0.5–1.1 milliseconds** for contexts ranging from 100,000 to 3 million characters, up to 437 times faster than the Hugging Face tokenizer. GPU-based full tokenization processes a 1-million-character request in 0.87 milliseconds.\n\n## The metric that matters is end-to-end latency\n\nAfter integration with vLLM, TokTier reduced median time to first token by **16%–34%** and lowered P99 latency by 23% under bursty traffic. Under a 50-millisecond P99 target, four repair CPU cores plus one GPU sustained 1,821 requests per second, while a 16-core stateless frontend saturated at 40 requests per second.\n\nThese numbers still need to be read carefully. The work is a preprint, and its evaluation is based on two agent ecosystems and a particular hardware configuration. A stateful tokenization service also introduces session routing, state consistency, GPU-utilization, and failure-recovery costs. That does not mean every chat API should add a GPU tokenizer.\n\nThe more important lesson is that the optimization target changes as the stack gets faster. Earlier work focused on model computation, KV-cache compression, and speculative decoding. Once those components improve, an ordinary CPU preprocessing step can become the longest pole in the request path.\n\nFor agent serving, inference optimization is no longer only about making the model calculate faster. It is about eliminating repeated work across the entire request pipeline. The next latency win may not come from a larger GPU or a new decoding trick; it may come from refusing to tokenize the same history twice.","toktier-stateful-tokenization-agent-serving","2026-07-31T17:56:30Z","2026-08-03T02:16:49.488442Z","2026-08-03T02:16:49.488464Z",true,"agent","\u002Fstatic\u002Farticles\u002Ftoktier-stateful-tokenization-agent-serving.webp",85,{"items":40},[41,46,51,56,61,66],{"id":42,"title":43,"news_slug":44,"published_at":45},"6b52b4a9-d567-46b8-99c1-e9c65ba59b16","SWE-Pruner Pro:ByteDance 让 Agent 自己当剪枝器,省 39% token 还涨分","swe-pruner-pro-bytedance","2026-07-25T12:00:00+00:00",{"id":47,"title":48,"news_slug":49,"published_at":50},"37aa0bc9-d135-444f-842e-0b40388d29e9","Qwen3.7-Max 原生兼容 Anthropic API 协议：Claude Code 现已可直接调用阿里模型","qwen3-7-max-anthropic-api-claude-code","2026-05-27T10:05:00+00:00",{"id":52,"title":53,"news_slug":54,"published_at":55},"57b691cc-476c-4427-8618-e29127654b34","AMD ROCm 7 原生支持 Qwen3-Coder-Next：单卡 256k 上下文打破推理硬件垄断","amd-rocm7-qwen3-coder-next-256k-mono","2026-05-25T16:10:00+00:00",{"id":57,"title":58,"news_slug":59,"published_at":60},"0d0e5ce8-fa18-4907-b811-2918ff8464e4","FlexSQL：小型LLM如何在Text-to-SQL任务上超越GPT-o3和DeepSeek-R1","flexsql-nus-text-to-sql-spider2-65pct-gpt-oss-120b","2026-05-05T10:15:00+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"69e52a42-19c7-4580-8c49-5446233fbdde","7B模型如何超越GPT-4o？ICLR Oral论文揭示AgentFlow流式训练新范式","agentflow-7b-icrl-oral-flow-grpo-14-9pct","2026-05-03T01:10:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00"]