Speech foundation models are being consolidated into a single network that handles generation, editing, separation and enhancement behind one natural-language instruction. That sounds elegant, but few-step distillation turns the accounting ugly: distribution matching distillation (DMD) works fine on generation, and when you move it to separation, it can train the student model to be worse than the original mixture.
The Route-DMD paper (arXiv:2611.08011) takes this failure apart in full. The work comes from a joint team across Shanghai Jiao Tong University, BIGAI, the Shanghai Innovation Institute, Nanyang Technological University and Tencent Hunyuan, and the teacher model used in the experiments is AuK, the unified speech foundation model open-sourced back in September (arXiv:2609.08936).
Pull generation away, and separation collapses
Start with the cost. Speech foundation models built on flow matching need 20–50 iterative steps per request, and classifier-free guidance (CFG) doubles the computation at every one of those steps. Few-step distillation targets exactly this: swap a multi-step teacher for a "student" that produces a result in a handful of steps.
The paper notes that prior speech distillation work was validated solely on generation tasks, and that distilling a speech foundation model which also supports editing and speech processing "remains unexplored". Route-DMD set out to open that road, and its first step was to break it: applying DMD directly destroys separation capability.
The ablation shows what the damage looks like. Without routing, the separation task's SI-SDRi (scale-invariant signal-to-distortion ratio improvement) falls from +6.6 dB all the way to -1.7 dB, while phone error rate (PER) climbs from 7.1% to 54.4%, with UTMOS and speaker similarity dropping in step.
The fault lies with the teacher, not the student
The mechanism the paper proposes is worth dwelling on. The supervision signal for distribution matching comes from the teacher scoring a noised state — but the teacher is only reliable near its training distribution. Once the student's early outputs are wrong, the noised version of those wrong outputs sits far away from where the teacher was trained, so the "real score" the teacher returns there is itself unreliable. Supervise a model with an unreliable signal and the more you train, the harder it collapses.
One control experiment nails the pathology down. Feed the teacher a noised ground-truth target instead of the noised student output, and DMD still runs on those samples without the collapse: SI-SDRi holds at +5.1 to +5.4 dB and PER stays within 7.0%–8.4%. The paper's conclusion is that the failure is located in the input the teacher receives.
The fix: swap "match the distribution" for "regress to the truth"
With the diagnosis in hand, the fix follows. The core insight is that a distribution-matching objective is not universal. Generation is described by a sentence and its answer is a distribution; separation, once the instruction is given, has a single correct answer — pulling a particular voice out of a mix has a ground truth. For that kind of task, a regression loss fits better than distribution matching.
So Route-DMD routes by task. Speaker separation and vocal separation are pulled out and trained with a clean-endpoint regression loss that fits the ground-truth latent directly; every other task keeps decoupled distribution matching. Training runs in two stages: first a consistency distillation initialization of 500 updates (teacher CFG at guidance scale 2), then the Route-DMD stage itself.
A second substitution replaces CFG with APG (adaptive projected guidance). At high guidance scales CFG over-extrapolates along the conditional direction and produces artifacts, and the student absorbs those artifacts along with everything else. APG operates in clean-prediction space, decomposing the conditional-minus-unconditional residual with respect to the conditional prediction, keeping the orthogonal component and down-weighting the parallel one by a coefficient η. The ablation is unambiguous: with CFG the student improves faster at first but saturates quickly, while with APG it keeps climbing and finishes with higher SIM and lower WER in both languages.
Two regressions the table shows and the abstract does not
The abstract says the 4-step student "matches" the 32-step teacher with a 4.5× end-to-end speedup. The speedup is real: on a single NVIDIA H20 the end-to-end real-time factor (RTF) drops from 0.4333 to 0.0958, which works out to 4.52× and matches the claimed 4.5×. Subjective evaluation moves the right way too, with naturalness MOS going from 4.23 ± 0.10 to 4.34 ± 0.10 and instruction adherence from 4.50 ± 0.09 to 4.57 ± 0.09 — the paper says both sit within the reported 95% confidence intervals of the teacher.
But there are regressions in the table, and you have to read them yourself. On Chinese instruction-following TTS (InstructTTSEval DSD-ZH), the 32-step AuK scores 83.4 while the 4-step Route-DMD student scores 78.8 — a 4.6 point drop, landing below the 81.1 of the Qwen3-TTS-VD baseline. On speech editing (MMAE-Speech), instruction following rate falls from 48.23 to 46.62 and consistency rate from 88.11 to 86.41; only exact match rate improves, from 12.44 to 13.85. Average WER on SeedTTS-Eval also rises, from 2.65 to 2.85, and on Libri2Mix separation WER climbs from 9.12 to 10.07.
The gains are concentrated elsewhere. Across all three speech processing benchmarks the student beats the teacher on both OVRL and UTMOS (CHiME-4 3.28→3.35, Libri2Mix 3.28→3.32, VCTK-SR 3.21→3.25), and on the hard Chinese subset test-zh-hard it takes the best SIM of 0.784.
So what
The value of this paper is probably not the 4.5×, but the judgment call it demonstrates and that is easy to skip: a multi-task unified model cannot be distilled with a single loss for everything.
On the training side AuK had already differentiated its post-training — human-feedback preference optimization for open-ended editing, reward-based reinforcement learning for speech generation. Distillation raises a different problem, because there the teacher has to act as a grader, and different tasks place very different demands on how reliable that grader must be. Distribution matching assumes the teacher can score states the student can actually reach; once a task has a naturally unique answer and erroneous outputs land far from the training distribution, that premise breaks.
The more actionable lesson is to read the table rather than the abstract. "4 steps match 32 steps" comes with a 4.6 point gap on Chinese instruction TTS and small regressions on the editing IFR/CR, while the gains concentrate on processing tasks and the hardest subsets. For anyone building a voice agent, that is a concrete planning question: if you plan to distill a unified speech model down to a few steps, decide first which of generation quality, instruction following, or processing capability you actually need to protect — they are not protected at the same time.
Paper and code: arXiv:2611.08011 · project page · GitHub repository