[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-f523f0fb-2b6f-4610-9eac-e3bdb3fc8cd9":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"f523f0fb-2b6f-4610-9eac-e3bdb3fc8cd9","tuetoken 把 LLM tokenizer 重做成 Rust 内核:7-40× 提速,丢进 vLLM 最多吃到 85% 吞吐","tokenization 是 LLM 栈里最无聊的部件——它每天被调用无数次,却一直躺在单线程、Python 包装的循环里没人优化。Vadim Borisov 团队的 tuetoken 把这件事重做了一遍:Rust 内核 + Python API,作为 `transformers.AutoTokenizer` 的 drop-in 替代,输出与参考实现 byte-identical,只是速度不同。\n\n核心改造两条。**一是把经典 BPE 的 O(n²) merger 换成线性版本**——长 hash、base64、随机 ID、压缩代码会让 chunk 变长,二次代价爆表,这正是 tokenizer 偶尔一次跑几分钟的元凶;线性 merger 让这类对抗性输入稳定在毫秒级。**二是真的多线程**。Rust 内核释放 GIL,192 核机器上单文档长文本吞吐从 35 MB\u002Fs 拉到 110 MB\u002Fs;tiktoken 仍卡在 16 MB\u002Fs 串行,HuggingFace tokenizers 自动多核也才 18 MB\u002Fs。整套 benchmark:p50 编码 16×、解码 18×、大 batch 5×、短 chat 30×,综合 7-40× 提速。\n\nvLLM 端收益更具体:长 prompt TTFT 显著下降(128 KB prompt 从几十毫秒压到个位数);高并发短请求吞吐最高 +85%;embedding 模型受益最明显(单前向,tokenization 在端到端占比反而最重)。覆盖 Llama \u002F Qwen \u002F Mistral \u002F Mixtral \u002F DeepSeek \u002F Gemma \u002F Phi \u002F GPT-OSS \u002F GLM \u002F Kimi,也原生吃 OpenAI 的 tiktoken。`pip install tuetoken`,改一行 import 就够,MIT 开源,无法复现则拒载而非悄悄近似——这点对生产很关键。","https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fvdmbrsv\u002Ftuetoken-fastest-llm-tokenizer","24d5c6c5-6573-4180-a1fd-f1459842d1af",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source","2026-07-04T18:09:49Z","2026-07-04T18:09:49.571694Z","2026-07-04T18:09:49.571706Z",true,"agent",4]