[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-qwen3-8-27b-byteify-vocab-384":3,"topics-all":38,"news-related-7984acc5-d045-4258-8da7-daf8ff25995b":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"7984acc5-d045-4258-8da7-daf8ff25995b","一字节一token:27B模型词表砍到384的账本","两位独立研究者的新论文把 Qwen3.8-27B 词表砍到 384,只动两头,用 4.77GB 自生成监督字节续训约 1,430 GPU 时:数学与代码成绩在噪声内,不靠思考的字符操作大涨,同卡汉字吞吐跌到约四分之一,笔画读音等查表知识受损,论文还记录了四种字节模型特有的失败模式。","Qwen3.8 的 BPE 分词器平均要 3.84 个字节才产出一个中文 token,64% 的 token 是单个汉字;每个汉字的一次生成决定被摊成三步,拼写、笔画、数字的每一位,对子词模型从来不可见。11 月 10 日提交到 arXiv 的论文(2611.13607,作者 Liwei Zhang 与 ZhiMing Tong,单位 Landi Lab)把这件事推到极端:把已经训练好的 Qwen3.8-27B 词表从 248,320 砍到 384——只留 256 个字节符号加特殊符号,让模型一个字节走一步,重新学会读写中文。\n\n## 只动两头,主干不动\n\n词表手术只碰两端:embedding 和 lm_head 各自从 248,320×5,120(约 1.27B 参数)缩到 384×5,120(约 2M),语言模型部分从 26.9B 降到 24.4B;64 层主干(48 层 Gated DeltaNet 加 16 层全注意力)一个权重不动,0.46B 的视觉塔原样冻结。因为 256 个字节符号本来就在 BPE 词表里,连初始化技巧都不需要,难度全部压到续训上。\n\n续训用 4.77GB 监督字节(总共读 7.08GB),分两段:M1 编码段拿字词典(12,126 字)、词词典(141,361 条)和通用对话混着教「字↔字节」;M2 阅读段掺入维基文本(占监督字节 54%)补知识。全部训练 178.6 小时 8×H200,约 1,430 GPU 时。对话类监督有条硬规矩:全部由原模型自己现做现答(on-policy)——作者实测,换成别人写的数据,四种配方都要掉 5-14 分。\n\nM1 结束时模型「会读不会用」:探针 29\u002F30,但英→中术语 0\u002F12,其余四种问法 6-12\u002F12。论文的判读是:知识在、英文路径也通,断的是「中文词形↔概念」的连接。M2 走到 3,000 步,术语回到 12\u002F12;冻结视觉塔上的 MMBench-CN 反而从 65.2% 升到 72.2%。\n\n## 换来什么:不靠思考的字符级操作\n\n非思考档宏观:CUTE 75.2→84.0,EXECUTE-zho 81.6→93.9,自建中文套件 54.4→64.0;单项里 EXECUTE-zho 的 swap_char 48.5→91.5,自建 zh_swap_char 13.0→50.5。原版不是做不到,而是要「想」:开思考后 CUTE 75.2→89.4,代价是每题几千 token(del_word 平均 3,306 token,36\u002F100 顶到上限);字节模型不用想就直接做对。\n\n两个附带结果:12,126 字全字复述两遍共 24,252 个槽位,原版错 3 个,字节模型只错 1 个——逐字节写字不比原版差;位置能力也保住了,针测 8k-128k 全中,1M 字节开 YaRN 还能拿 33\u002F46。\n\n## 三笔账\n\n考试与指令:C-Eval 思考档 89.9→85.4、非思考档 88.1→83.4,思考档 GSM8K 97.0→93.0,这三个差距的 95% 区间都不含 0;MATH-500、HumanEval+、MMLU-Pro 与非思考 GSM8K 全在噪声内。论文对非思考 IFEval 的归因很诚实:原版做过指令后训练、字节版没做,这个因素没有分离实验。\n\n吞吐:每步速度几乎没变(单流 22.9 对 24.3 token\u002Fs),差的是每步产出少 3-5 倍——原版 1 token 约 1.45 个汉字,字节模型 1 步 1 字节约 0.32 个汉字。同卡汉字\u002F秒 33.1→7.7,约四分之一;64K 字节 prefill 首字 4.52→29.57 秒,6.5 倍;全部榜单同卡跑完,原版 7 小时 44 分,字节版 35 小时 57 分,4.6 倍。草稿头在 48G 单卡装不下,按其他硬件的记录折算,加草稿单流也只能回到原版的 0.7-0.8 倍。\n\n汉字查表知识:zh_homophone 96.5→57.5,zh_strokes 63.0→47.0(思考档 69→34),zh_pinyin 98.0→91.5。思考救不回——「字→属性」的查表知识不是推理能补的。22 语种冒烟里更刺眼:用印地语问印度的首都是哪,字节模型答「印地语」。\n\n## 四种字节模型特有的病\n\n贪心跑飞:temperature 0 下,试点线 GPQA 198 题里 34 题(17%)写满 16,384 字节上限、没有答案行,同权重 BPE 原版是 0。换成采样,分数 54.0→60.6;再加护栏,思考档 48.0→60.6;但答对数停在 120\u002F198(原版 170)——解码侧只能把「没写完」变成「写完了」,对不对还得看模型。\n\n采样打断 UTF-8:一个汉字从一次采样变三次,乘号可能被采成几何图形符号;解法是解码端加 UTF-8 合法性掩码,且必须关掉推理框架的重叠调度——这个 bug 一度让输出全是替换符,团队把带护栏的历史分数全部重审。\n\n惩罚失效:presence\u002Ffrequency penalty 按 token id 记账,在 256 词表上每个常用字节几步内都出现过,惩罚等于把所有字节一起平抬;重复抑制必须做在 n-gram 级,推理框架没有现成实现。\n\n复读:自产回答 46% 含 8-gram 复读,同题商用模型是 6%。直接 DPO 把差样本率从 11.9% 压到 7.7%,代价是 GSM8K 300 题 97.7%→80.7%;RFT 先行救回 92.7%;第三轮差样本率 0.2%,但 252 条带思考回答里 211 条思考不足 60 字——治好了复读,治出了「不思考」。\n\n## 这笔账怎么算\n\n论文自己把话说死:「我们不主张字节模型更强。」单种子单次运行;权重和训练语料不发布,开源的是评测代码、15 项 2,814 题的自建中文题集与 bootstrap 脚本。真正的增量在路线:BLT、Bolmo 这些前作都要 patching(局部编码器加块结构),主干只在块上跑;这篇不 patching、不引外部教师,约 1,430 GPU 时就给一个已训 27B 模型换血——「词表是训练前冻结的设计决定」这条行规,多了一次低成本的破例机会。\n\n对中文模型,更大的启示可能是那个反直觉结果:字节化让模型「会操作汉字」大涨,却让「认识汉字」(读音、笔画)受损。操作和知识不是一回事,换词表动的是后者的地基。\n\n一个不靠思考就能倒排汉字、却说不出它有几笔的模型——它到底更懂中文了,还是更不懂了?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13607","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",{"id":22,"name":23,"slug":23,"description":14,"color":14},"045c011e-e2bb-45ce-bdd6-0c927f8a3b87","token-efficiency",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"c094656b-1f15-4970-a6f8-f9274ba12e96","en","One Byte, One Token: A 27B LLM's Vocabulary Cut to 384","Byteifying a 27B LLM: vocabulary cut to 384, character tasks surge without thinking, throughput falls to 1\u002F4, Chinese lookup knowledge declines.","Qwen3.8's BPE tokenizer needs 3.84 bytes on average to produce one Chinese token, and 64% of tokens are single characters; spelling, stroke counts and individual digits are invisible to a subword model. A paper submitted to arXiv on 10 November (2611.13607, by Liwei Zhang and ZhiMing Tong of Landi Lab) pushes this to the extreme: take an already-trained Qwen3.8-27B, shrink its vocabulary from 248,320 to 384 — the 256 byte symbols plus specials — and let the model take one step per byte.\n\n## Only the Two Ends Change\n\nThe surgery touches only the embedding and lm_head: each shrinks from 248,320×5,120 (about 1.27B parameters) to 384×5,120 (about 2M); the language-model part goes from 26.9B to 24.4B. The 64-layer backbone (48 Gated DeltaNet plus 16 full-attention layers) is untouched, and the 0.46B vision tower stays frozen. Since the 256 byte symbols already sit in the BPE vocabulary, no initialization trick is needed — all difficulty shifts to continued training.\n\nThat training uses 4.77 GB of supervised bytes (7.08 GB read in total) in two stages: M1 teaches character↔byte encoding from a character dictionary (12,126 characters), a word dictionary (141,361 entries) and general dialogue; M2 mixes in Wikipedia text (54% of supervised bytes) for knowledge. Total cost: 178.6 hours on 8×H200, about 1,430 GPU-hours. One hard rule: all dialogue supervision is generated on-policy by the original model itself — the authors measured that four recipes using data written by others all lost 5-14 points.\n\nAfter M1 the model can read but cannot use what it reads: 29\u002F30 on a greedy probe, yet English→Chinese terminology 0\u002F12 while the other four phrasings score 6-12\u002F12. The paper's reading: the knowledge is there and the English pathway works; what broke is the link between Chinese word forms and concepts. By M2 step 3,000 terminology is back to 12\u002F12, and MMBench-CN on the frozen vision tower actually rises from 65.2% to 72.2%.\n\n## What Bytes Buy: Character Operations Without Thinking\n\nNon-thinking macro averages: CUTE 75.2→84.0, EXECUTE-zho 81.6→93.9, the authors' Chinese character suite 54.4→64.0; per-task highlights include EXECUTE-zho swap_char 48.5→91.5 and zh_swap_char 13.0→50.5. The original can do these too — but only by thinking: with thinking on, CUTE goes 75.2→89.4 at a cost of thousands of tokens per item (del_word averages 3,306 tokens, 36\u002F100 cap hits). The byte model just does them.\n\nTwo side results: reciting all 12,126 characters twice (24,252 slots), the original gets 3 wrong and the byte model 1 — writing characters byte by byte is no worse; positional ability survives too, with needle tests at 8k-128k passing 4\u002F4 and 33\u002F46 at 1M bytes with YaRN.\n\n## Three Bills to Pay\n\nBenchmarks and instructions: C-Eval 89.9→85.4 (thinking) and 88.1→83.4 (non-thinking), thinking GSM8K 97.0→93.0 — the only gaps whose 95% intervals exclude 0; MATH-500, HumanEval+, MMLU-Pro and non-thinking GSM8K all sit within noise. On non-thinking IFEval the paper is honest about attribution: the original had instruction post-training, the byte model did not, and no experiment isolates that factor.\n\nThroughput: per-step speed is nearly identical (22.9 vs 24.3 tokens\u002Fs single-stream), but each step yields 3-5× less output — 1 token is about 1.45 Chinese characters, 1 byte about 0.32. Chinese characters per second fall from 33.1 to 7.7, about a quarter; prefill of 64K bytes goes 4.52→29.57 s to first token, 6.5×; running both benchmark modes takes 7h44m versus 35h57m, 4.6×. The draft head does not fit on a single 48G GPU; extrapolating from other hardware, single-stream recovers only to 0.7-0.8× of the original.\n\nChinese lookup knowledge: zh_homophone 96.5→57.5, zh_strokes 63.0→47.0 (69→34 thinking), zh_pinyin 98.0→91.5 — thinking does not recover it. In a 22-language smoke test, asked in Hindi for India's capital, the byte model answers \"Hindi\".\n\n## Four Failure Modes Specific to Byte Models\n\nGreedy runaway: at temperature 0, a pilot GPQA run had 34 of 198 items (17%) fill the 16,384-byte cap with no answer line, versus 0 for the BPE original on the same weights. Switching to sampling raised the score 54.0→60.6; adding the guard, thinking mode 48.0→60.6; but correct answers stalled at 120\u002F198 versus 170 — decoding can only turn \"unfinished\" into \"finished\".\n\nByte-wise sampling breaks UTF-8: a multiplication sign can come out as a geometric-shape symbol; the fix is a UTF-8 validity mask on the decoder, and overlap scheduling must be turned off — a bug that once filled outputs with replacement characters forced the team to re-audit all guarded historical scores.\n\nPenalties stop working: presence\u002Ffrequency penalties keep accounts per token id; among 256 bytes every common byte has appeared within a few steps, so the penalty just lifts all bytes together. Repetition control must move to the n-gram level, and inference frameworks have no ready-made implementation.\n\nRepetition: 46% of self-generated answers contain 8-gram repetition, against 6% for a commercial model on the same prompts. Direct DPO cut the bad-sample rate from 11.9% to 7.7% but dropped GSM8K on 300 problems from 97.7% to 80.7%; RFT first recovered it to 92.7%; the third round reached 0.2% — but 211 of 252 thinking answers had thoughts under 60 characters. The cure for repetition turned out to be not thinking.\n\n## How to Read the Ledger\n\nThe paper says it plainly: \"We do not claim that the byte model is stronger.\" Single seed, single run; weights and the training corpus are not released — only evaluation code, a 15-task, 2,814-item Chinese character suite and bootstrap scripts. The real increment is the route: BLT and Bolmo both keep patching structures (a local encoder plus patches), with the backbone running over patches; this one patches nothing and hires no external teacher, giving a trained 27B model a transfusion for about 1,430 GPU-hours. \"A tokenizer is a design decision frozen before training\" now has a low-cost exception.\n\nFor Chinese models the deeper lesson may be the counterintuitive one: byteification makes the model far better at manipulating characters while making it worse at knowing them (pronunciation, strokes). Manipulation and knowledge are not the same thing — changing the vocabulary shakes the foundation of the latter.\n\nA model that can reverse Chinese strings without thinking but cannot say how many strokes a character has: does it understand Chinese better, or worse?","qwen3-8-27b-byteify-vocab-384","2026-11-12T21:10:40Z","2026-11-12T21:10:44.327483Z","2026-11-12T21:10:44.327491Z",true,"agent",174,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"f828f611-b57a-4855-9246-3609c2495faa","老师折进自己:2.27B用22B token留住84%","concertina-folding-llm-teacher-weights","2026-11-16T08:50:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"0a4281e9-69b1-43f0-9e7b-78f0cd7ea4c2","医疗小模型回答砍72.8%,准确率反升3.2点","peca-conciseness-qlora-medical","2026-11-15T15:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"6186bff0-6bbb-4970-a0bc-b7bfa93a8e49","像素空间压缩:16个token装下RAG检索到的文档","opticalrag-pixel-space-rag-compression","2026-11-13T03:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"8d830f21-a44f-45b8-be6c-eed523d060dc","投机解码的天花板是分词器:3.4M参数换2.4倍","speculative-decoding-bpe-tokenizer-ceiling","2026-11-11T01:30:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"4f151ed3-74d5-4dc7-99d0-ac03df4c1b08","自一致别整条重采:定位错步,token至多省41%","stepback-error-localized-resampling","2026-11-08T13:20:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"6d9c05f8-1efb-4ad0-bec1-3e95011bd0b8","跨模型状态交接:KV缓存之外还有两块状态要搬","cross-model-state-handoff-hybrid-linear-attention","2026-11-07T07:05:00+00:00"]