Tencent open-sourced WeDLM, a diffusion language model (dLLM) that achieves 3-6× speedup over standard vLLM, while using standard causal attention. The standout: the speedup is achieved without modifying the underlying attention mechanism, making WeDLM compatible with all existing vLLM optimizations.
The "dLLM + causal attention" insight: most dLLMs require a custom attention mechanism (e.g., bidirectional attention), which is incompatible with vLLM's standard causal attention optimizations (e.g., PagedAttention, continuous batching). WeDLM's fix: keep the standard causal attention, but use a "parallel-decoding" schedule that generates multiple tokens in parallel. The result is dLLM-like parallelism with vLLM-like efficiency.
The technical details: WeDLM uses a "block-wise parallel decoding" schedule. The input is split into blocks of N tokens, and the model generates all N tokens in parallel within a block. The blocks are generated sequentially (block 1, then block 2, etc.), but within each block, all tokens are generated in parallel. The result is O(n/N) sequential steps, where N is the block size.
The benchmark: WeDLM hits 3-6× speedup over standard vLLM (which is already optimized for AR generation). The quality is comparable to AR baselines on MMLU, HumanEval, and GSM8k. The model is fully open-sourced, with vLLM-compatible code.
The bigger takeaway: "dLLM + vLLM" is the right combination for production diffusion LLMs. The "custom attention" requirement of previous dLLMs was a major barrier to adoption, and WeDLM's "causal attention compatibility" opens up dLLM to the existing vLLM ecosystem.