Is next-token autoregression the only way to build a language model? Researchers from ByteDance Seed's Seedance team, with the University of Hong Kong, Peking University and Renmin University of China, just posted a 47-page arXiv paper, Cola2, that gives the alternative route its most systematic engineering treatment yet: compress text into continuous semantic latents first, then let a diffusion decoder do the writing. Paper: https://arxiv.org/abs/2610.31305.
A two-level pipeline
Cola2's core is a compression-generation division of labor. Stage one trains the compression encoder: text is dynamically segmented into content-aware semantic spans, latents are allocated per span, and the compression rate is held near target by an auxiliary-loss-free negative-feedback controller, so it never fights representation learning. Stage two introduces a roughly 4B-parameter latent DiT: Flow Matching fits the latent distribution, with its gradient decoupled from the encoder via stop-gradient, while token-prediction gradients keep flowing through the latents to update the encoder, so continuous representation and discrete generation co-evolve. Stage three continues training the DiT alone.
The decoder is a block-causal masked diffusion model (MDM) trained on a mixture of dropped, noisy and clean latent conditions at a 0.4/0.4/0.2 ratio; at inference, classifier-free guidance tunes the balance between faithful semantic reconstruction and autonomous generation. Ablations show shuffling the latents costs 12.2 points, evidence the decoder really uses the aligned semantics. The inference loop is propose, decode, re-encode: the DiT proposes a latent block, the decoder realizes tokens, and the generated text is re-encoded under a fixed span layout. DiT denoising saturates at about eight steps.
The 346B-token report card
The largest pretraining run initializes the causal decoder from Qwen3-4B-Base plus a roughly 4B latent DiT. Using only 346.03B source tokens, and explicitly unconverged, it reaches a 60.10 average over eight benchmarks, 6.15 points behind Llama-3-8B (66.25, reported budget 15T+ tokens), while surpassing it on HellaSwag (41.75 vs 39.21) and StoryCloze (80.42 vs 79.49). The paper stresses the token budgets differ by more than 43 times. The project page's full table: Qwen2.5-7B (18T) averages 75.08, Qwen3-8B-Base (about 36T) 75.87, LLaDA-8B-Base (2.3T) 72.11. The first author also announced on Hugging Face that an 8B model will be open-sourced in the coming weeks; the project page's Code and Model links still read "Soon".
The cold water
First, the 43x efficiency figure is the paper's own framing, and the decoder stands on Qwen3-4B-Base's shoulders — the authors themselves attribute results to effective transfer of the Qwen-initialized causal decoder's generation capability. This is not a from-scratch 346B miracle. Second, compression cuts the latent prior's linear and attention FLOPs in theory, but end-to-end speed still depends on the decoder, CFG branches and hardware utilization; the authors explicitly call for infrastructure co-design — specialized kernels, cache management, parallel scheduling — to turn theoretical savings into real speedups. Third, the ablations are honest: generative supervision lifts accuracy from 15.7% to 26.7%, encoder-decoder co-training adds 5.9 points, but the compressed Cola2-MDM converges more slowly and only starts closing the gap with LLaDA after about 150B tokens.
So what: Cola2 proves the continuous-latent-plus-discrete-generation route scales and competes, not that it has won. The judge of paradigm contests has always been cost per token; the real watershed arrives when the 8B weights land and third parties reproduce the numbers.