Diffusion language models want to challenge autoregressive generation, but they are stuck on a structural bottleneck: when decoding in parallel, each token is sampled independently from its own marginal distribution, severing the statistical dependencies among tokens decoded together. On a globally constrained task like Sudoku, two cells in the same row have no idea the other exists, so accuracy stalls. Continuous diffusion LMs sidestep this by planning all tokens jointly in a shared continuous latent — but their denoiser only sees the latent itself, and nothing ties it to a valid token configuration until the very last step. HC-DLM (Hierarchical Continuous Diffusion Language Models), submitted to arXiv on October 1 by a UIUC team, stitches the two routes together: discrete token generation and a continuous latent trajectory are coupled into a single denoising process, with a training objective derived from a variational bound on token likelihood.

The latent as the only persistent draft, tokens correct it every step

The key design makes the latent the only persistent generative state: at every step the model reads a token draft k_t out of the latent x_t, and that draft guides the next latent update. Unlike recent hybrid methods that attach continuous context to a self-contained discrete chain, there is no separate discrete chain here — tokens are readouts of the latent, yet they serve as its anchor. On the training side, one variational bound splits into three terms: a reconstruction term for token readout, a token-guided flow-matching term for the latent denoiser, and an entropy regularizer that keeps the encoder from collapsing, all trained end to end.

Numbers: the best in the diffusion camp at matched scale

The project page lines up four design families: discrete diffusion (MDM, LLaDA-style) leaves tokens independent within a step; continuous diffusion (Diffusion-LM, Plaid) decodes only at t = 0; hybrids (CCDD) couple via conditioning. HC-DLM keeps both strengths — tokens are planned jointly in the latent and corrected by tokens at each step. Per the project page:

  • Hard Sudoku (6M params): HC-DLM reaches 72.41%, +1.68 points over CCDD and +22.53 over MDM (top-prob. margin); a 7x larger autoregressive model manages only 32.57% here.
  • Countdown CD5 (6M params): 37.52%, +12.17 points over CCDD; the lead widens with planning depth — +3.2 on CD4, +12.17 on CD5.
  • LM1B generative perplexity (118M params): 75.5, below Plaid's 77.3, LangFlow's 92.2, and MDM's 103.9.

The cold water is on the same page: on Easy Sudoku, CCDD's 94.65% still edges out HC-DLM's 94.21%; on LM1B, an autoregressive Transformer (108M) at 66.7 remains clearly below every diffusion option. HC-DLM's "lead" holds only within the diffusion camp — the autoregressive wall is yet to crack.

Ablations: neither level works alone

Remove token guidance and the model degenerates into a plain latent diffusion model: Hard Sudoku drops from 72.41% to 24.74%. Remove the latent and it becomes a purely discrete chain: 49.88%. The forward-noise choice matters too: absorbing-state (mask-style) noise tops out at 48.25%, while uniform-state corruption with parallel updates reaches 72.41% — a uniformly corrupted sequence is still a complete hypothesis, whereas masks carry no scaffold information. Two process curves deserve attention: the accuracy of intermediate predictions climbs past 90% before t/T = 0.6 (the no-guidance variant plateaus near 57%); and when denoising steps shrink from 100 to 10, HC-DLM holds the 52%-76% range while MDM falls from 56% to 20% — the few-step regime is exactly where diffusion's speed advantage over autoregressive decoding lives, so this curve is what ties the method to practical value.

So what

Over the past year the diffusion-LM conversation has moved from "can it generate" to "how to preserve dependencies under parallel decoding" — Flash-dLLM attacked it with training-free acceleration, dQwen3.5 with adapter breakpoints, and HC-DLM restructures the generative state itself. For inference teams, the value is not the 6M toy scale but the validated idea that joint planning and step-wise anchoring can coexist without collapsing at few steps. Whether conclusions at 6M/118M transfer to frontier scale is left to future work — but the paradigm of "the latent drafts, tokens correct every step" earns a slot on your reading list.

Reference: arXiv:2610.02193 · Project page