The economics of diffusion language models (DLMs) have always looked bad: adapting a pretrained autoregressive (AR) model instead of training from scratch is the accepted cost-saving route. But a six-person team at UT Austin noticed a mismatch the entire field had been steering around — nearly all AR-to-DLM adaptation research starts from a full-attention transformer, while real-world frontier open models abandoned pure full attention long ago.

Hybrid backbones are the reality, and they resist DLM adaptation

About three-quarters of Qwen3.5's sequence-processing layers are recurrent Gated DeltaNet (GDN) layers; only a quarter are attention. Recurrent layers are structurally causal — state flows left to right, and "looking both ways" over the whole sequence is architecturally off the table. That is exactly why past adaptation work defaulted to full-attention backbones: bidirectionalizing attention is a masking change; an RNN is not.

The team's fix is counterintuitively simple: don't bidirectionalize the RNN. dQwen3.5 only makes its attention layers bidirectional, leaves the GDN layers causal as-is, and applies token shifting — the hidden state at position k still predicts the token at position k+1, preserving the readout alignment from AR pretraining.

They adapted Qwen3.5 at 0.8B/2B/4B/9B scales under one shared recipe, calling the result the dQwen3.5 family; a full-attention Qwen3-1.7B was adapted as a trunk-matched control. Training ran at 50B and 100B token budgets, batch 512 sequences × 4096 tokens, bf16 precision.

The numbers: half the tokens, and 50B beats 580B

Two layers of results are worth remembering:

  • Efficiency: against the trunk-matched full-attention control, the hybrid dQwen3.5-2B reaches a given training loss in about half the tokens.
  • Effectiveness: dQwen3.5-9B after only 50B adaptation tokens takes the highest score on 4 of 7 benchmarks against Dream-7B (580B tokens), Dream-Coder-7B (322B), and LLaDA-8B (2.3T, trained from scratch); at 2B, 50B tokens beat CoDA (200B tokens) on 6 of 7 benchmarks.

The honest flip side

On the knowledge-vs-code balance, adaptation has a cost: on the 0.8B model, original Qwen3.5 scores 50.31 MMLU while the 50B-token adapted version manages 32.62 — knowledge-heavy capability drops sharply, while code benchmarks actually rise (HumanEval 22.56 → 28.05). "Longer is better" also fails: going from 50B to 100B improves 5 of 7 benchmarks at 0.8B but only 1 of 7 at 9B. And the causal RNN structure does not kill the DLM's soul: dQwen3.5 matches full-attention DLMs in any-order decoding behavior (local AR-ness of 0.58–0.66, inside the same range as every other DLM evaluated), and under parallel decoding it retains more HumanEval performance at higher speedup ratios than comparable-trunk peers.

Open weights and engineering details

The full dQwen3.5 family (0.75B/1.88B/4.21B/8.95B hybrids plus a 1.72B full-attention control) is open on Hugging Face under the UT-IFML org, with code on GitHub (AntonXue/dQwen). The bar is not low: transformers>=5.13 and flash-linear-attention 0.5.1 required; inference decodes the whole canvas at once, committing positions above confidence threshold tau=0.9, or uses block_length=32 for block-wise left-to-right decoding. At writing time the 9B model shows 543 downloads last month — a current-state number, for reference only.

So what

The real contribution here is not benchmark points but dragging DLM adaptation research from the "full-attention habit" back to the hybrid backbones that actually dominate shipping models. As Jamba, MiniMax, and Qwen all migrate toward attention+RNN hybrids, a DLM community that only adapts full-attention models is building on a route that no longer exists. That dead end is now connected — with open weights at four sizes and a reproducible recipe, this is the most practical starting point for any team that wants to own a DLM cheaply.

(References: arXiv:2609.20751 · UT-IFML/dQwen3.5-9B-Base)