Meta FAIR byte-distillation study: End-Of-Token asymptotically beats token distillation by 4%, using only 1/6 of the data

Why byte distillation looks slow but wins asymptotically

Byte-level distillation has long been outshone by token-level distillation at modest compute budgets. A new paper from Meta FAIR and the University of Washington, Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models (arXiv:2609.12303), flips the script: at scale, the byte student can outperform its token counterpart.

The team — led by Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz, Margaret Li, Mike Lewis, Luke Zettlemoyer and Srinivasan Iyer — fixed layer-parameter counts (about 1.28B layer params for byte variants, 1.81B for token variants) and swept training up to roughly one trillion bytes. They crossed two axes simultaneously: tokenization (tokens, bytes, bytes with end-of-token) and objective (distillation vs. cross-entropy). To compare apples to apples they had to convert the teacher's token logits into byte logits in a single forward pass, introducing two variants: Marginalize-It (approximate, redistributes dropped probability mass) and End-Of-Token (exact, appends an <eot> marker so no mass is lost).

Across eight benchmarks spanning multiple-choice QA (ARC-Easy, ARC-Challenge, HellaSwag, PIQA), language generation (MBPP, Natural Questions) and machine translation (Flores), the token-1B curves lead at low FLOPs but saturate quickly. The byte curves start worse but climb at a steeper slope. Fitting power laws on downstream error vs. validation BPB, the authors extrapolate that distilled End-Of-Token-1B beats distilled Token-1B asymptotically by up to 4% on averaged downstream accuracy, and beats Marginalize-It distillation by about 1.9%.

Two numbers matter most for production pipelines. First, End-Of-Token distillation matches the final Token-distilled performance using only one-sixth of the training data — a data-cost win on top of a compute win. Second, by shrinking the vocabulary from roughly 100K tokens to about 256 bytes, the distillation pipeline no longer needs top-k truncation when dumping logits, dropping logit storage cost to roughly one-fifth.

The paper also benchmarks the asymptotic prediction against open-weight models: distilled End-Of-Token-1B would exceed Llama 3.2-1B by up to 6.5%, Gemma-3-1B-pt by 8.1%, and Gemma 2B by 2.1% on averaged tasks. These are extrapolation ceilings from fitted scaling laws, not numbers measured on today's checkpoints, so read them as "what the trend predicts at convergence" rather than "what the model already does."

The methodological takeaway is sharp. Validation BPB, the lingua franca of pretraining, becomes misleading across tokenization schemes and objectives: the same BPB maps to very different downstream accuracy depending on whether the student operates on tokens or bytes. Picking a student on validation loss alone is no longer safe. Pair it with a downstream-error scaling law before signing off.

The second lesson is engineering-flavored. Logit-storage cost in distillation is widely underpriced. With a 100K-token vocabulary, even top-100 truncation stores 100 floats per position; with a 256-byte vocabulary you can keep the full distribution. Meta FAIR's result is in effect saying the bottleneck for "can we run a really big distillation run" is shifting from FLOPs to logit I/O.

For Chinese teams training small distilled models or doing byte-level multimodal pretraining (e.g. unifying text, image, audio at the byte level), this paper is a directly usable scaling-law template. Byte models are not winning the next quarter, but the ceiling is higher and the data bill is smaller — a combination that compounds.