Qwen3.8's BPE tokenizer needs 3.84 bytes on average to produce one Chinese token, and 64% of tokens are single characters; spelling, stroke counts and individual digits are invisible to a subword model. A paper submitted to arXiv on 10 November (2611.13607, by Liwei Zhang and ZhiMing Tong of Landi Lab) pushes this to the extreme: take an already-trained Qwen3.8-27B, shrink its vocabulary from 248,320 to 384 — the 256 byte symbols plus specials — and let the model take one step per byte.

Only the Two Ends Change

The surgery touches only the embedding and lm_head: each shrinks from 248,320×5,120 (about 1.27B parameters) to 384×5,120 (about 2M); the language-model part goes from 26.9B to 24.4B. The 64-layer backbone (48 Gated DeltaNet plus 16 full-attention layers) is untouched, and the 0.46B vision tower stays frozen. Since the 256 byte symbols already sit in the BPE vocabulary, no initialization trick is needed — all difficulty shifts to continued training.

That training uses 4.77 GB of supervised bytes (7.08 GB read in total) in two stages: M1 teaches character↔byte encoding from a character dictionary (12,126 characters), a word dictionary (141,361 entries) and general dialogue; M2 mixes in Wikipedia text (54% of supervised bytes) for knowledge. Total cost: 178.6 hours on 8×H200, about 1,430 GPU-hours. One hard rule: all dialogue supervision is generated on-policy by the original model itself — the authors measured that four recipes using data written by others all lost 5-14 points.

After M1 the model can read but cannot use what it reads: 29/30 on a greedy probe, yet English→Chinese terminology 0/12 while the other four phrasings score 6-12/12. The paper's reading: the knowledge is there and the English pathway works; what broke is the link between Chinese word forms and concepts. By M2 step 3,000 terminology is back to 12/12, and MMBench-CN on the frozen vision tower actually rises from 65.2% to 72.2%.

What Bytes Buy: Character Operations Without Thinking

Non-thinking macro averages: CUTE 75.2→84.0, EXECUTE-zho 81.6→93.9, the authors' Chinese character suite 54.4→64.0; per-task highlights include EXECUTE-zho swap_char 48.5→91.5 and zh_swap_char 13.0→50.5. The original can do these too — but only by thinking: with thinking on, CUTE goes 75.2→89.4 at a cost of thousands of tokens per item (del_word averages 3,306 tokens, 36/100 cap hits). The byte model just does them.

Two side results: reciting all 12,126 characters twice (24,252 slots), the original gets 3 wrong and the byte model 1 — writing characters byte by byte is no worse; positional ability survives too, with needle tests at 8k-128k passing 4/4 and 33/46 at 1M bytes with YaRN.

Three Bills to Pay

Benchmarks and instructions: C-Eval 89.9→85.4 (thinking) and 88.1→83.4 (non-thinking), thinking GSM8K 97.0→93.0 — the only gaps whose 95% intervals exclude 0; MATH-500, HumanEval+, MMLU-Pro and non-thinking GSM8K all sit within noise. On non-thinking IFEval the paper is honest about attribution: the original had instruction post-training, the byte model did not, and no experiment isolates that factor.

Throughput: per-step speed is nearly identical (22.9 vs 24.3 tokens/s single-stream), but each step yields 3-5× less output — 1 token is about 1.45 Chinese characters, 1 byte about 0.32. Chinese characters per second fall from 33.1 to 7.7, about a quarter; prefill of 64K bytes goes 4.52→29.57 s to first token, 6.5×; running both benchmark modes takes 7h44m versus 35h57m, 4.6×. The draft head does not fit on a single 48G GPU; extrapolating from other hardware, single-stream recovers only to 0.7-0.8× of the original.

Chinese lookup knowledge: zh_homophone 96.5→57.5, zh_strokes 63.0→47.0 (69→34 thinking), zh_pinyin 98.0→91.5 — thinking does not recover it. In a 22-language smoke test, asked in Hindi for India's capital, the byte model answers "Hindi".

Four Failure Modes Specific to Byte Models

Greedy runaway: at temperature 0, a pilot GPQA run had 34 of 198 items (17%) fill the 16,384-byte cap with no answer line, versus 0 for the BPE original on the same weights. Switching to sampling raised the score 54.0→60.6; adding the guard, thinking mode 48.0→60.6; but correct answers stalled at 120/198 versus 170 — decoding can only turn "unfinished" into "finished".

Byte-wise sampling breaks UTF-8: a multiplication sign can come out as a geometric-shape symbol; the fix is a UTF-8 validity mask on the decoder, and overlap scheduling must be turned off — a bug that once filled outputs with replacement characters forced the team to re-audit all guarded historical scores.

Penalties stop working: presence/frequency penalties keep accounts per token id; among 256 bytes every common byte has appeared within a few steps, so the penalty just lifts all bytes together. Repetition control must move to the n-gram level, and inference frameworks have no ready-made implementation.

Repetition: 46% of self-generated answers contain 8-gram repetition, against 6% for a commercial model on the same prompts. Direct DPO cut the bad-sample rate from 11.9% to 7.7% but dropped GSM8K on 300 problems from 97.7% to 80.7%; RFT first recovered it to 92.7%; the third round reached 0.2% — but 211 of 252 thinking answers had thoughts under 60 characters. The cure for repetition turned out to be not thinking.

How to Read the Ledger

The paper says it plainly: "We do not claim that the byte model is stronger." Single seed, single run; weights and the training corpus are not released — only evaluation code, a 15-task, 2,814-item Chinese character suite and bootstrap scripts. The real increment is the route: BLT and Bolmo both keep patching structures (a local encoder plus patches), with the backbone running over patches; this one patches nothing and hires no external teacher, giving a trained 27B model a transfusion for about 1,430 GPU-hours. "A tokenizer is a design decision frozen before training" now has a low-cost exception.

For Chinese models the deeper lesson may be the counterintuitive one: byteification makes the model far better at manipulating characters while making it worse at knowing them (pronunciation, strokes). Manipulation and knowledge are not the same thing — changing the vocabulary shakes the foundation of the latter.

A model that can reverse Chinese strings without thinking but cannot say how many strokes a character has: does it understand Chinese better, or worse?