"Who said this line" used to be bolted onto transcripts by a cascaded pipeline: an ASR model produces the words, then a separate diarization system aligns speakers. Long meetings mean misaligned labels and lost overlapping speech. Alibaba Token Foundry's Qwen team posted the Qwen-Audio-3.1-ASR technical report to arXiv on October 20, and the approach merges the two steps: the model directly emits structured transcripts with speaker labels and timestamps, and overlapping speech survives in the output structure.

Diarization moves into the model itself

The report frames this generation as two capability extensions: speaker-diarized transcription and general audio understanding. The former outputs who, when, and what, keeping speaker labels consistent across turns; the latter ships in the Next variant and covers captioning across speech, environmental sound and music, plus temporal grounding, sound event detection and audio question answering. The family still supports 30 languages and 16 Chinese dialects, and retains the instruction-controlled controls of the previous generation: historical-context conditioning, two-tier P0/P1 hotword customization, and native transcription polishing.

Deployment splits into five variants: asynchronous file transcription, low-latency streaming, context-aware voice input, non-streaming recognition, and Next. The capability matrix has explicit trade-offs — the streaming and Message ASR profiles do not support diarization or audio understanding; those live only in the non-streaming variants.

20 million hours and a six-stage training stack

Training runs six stages: audio encoder pretraining, audio LLM pretraining, cooldown adaptation, SFT, reinforcement learning, and multi-teacher on-policy distillation. The encoder is first pretrained with BEST-RQ self-supervision on unlabeled speech spanning AI, biotech, e-commerce, education, entertainment, finance and mobility; some Transformer layers initialize from a pretrained Qwen3 LLM, and the non-streaming encoder swaps causal attention for bidirectional. The audio LLM decoder initializes from the language-model component of Qwen on a MoE backbone, pretrained on roughly 20 million hours of multilingual data with the encoder frozen; the cooldown stage then rebalances languages and dialects on about one million higher-quality hours, re-transcribing with VibeVoice and dropping samples with word error rate above 0.3.

Two engineering details are worth noting: speaker-ID tokens account for only 2.90% of supervised tokens and 1.97% of the unweighted loss, so the team applied loss reweighting; timestamp precision is quantized relative to 1 ms, which speeds up DER and cpWER convergence. The RL stage uses the FunVerl framework with rewards including 1−WER, asymmetric hotword rewards, cpWER, and time-consistent tcpWER; distillation routes examples to three audio teachers plus a Qwen3.5 text-and-dialogue teacher.

Benchmarks: dialects are the home turf, MMSU not won

Attribution first: all numbers below come from the official technical report; Panel B results come from the team's unified API evaluation pipeline, with some baselines taken from public reports and some self-measured. On multilingual ASR, the model posts the lowest error rate among compared systems on all seven GigaSpeechBench East/Southeast Asian languages, macro-average 15.09% versus 22.50% for the previous generation; the comparison set includes Doubao-ASR, Tencent Cloud ASR, Gemini 3.1 Pro, GPT-4o Transcribe, Azure MAI-Transcribe-2, and Muse Voice Transcribe-1.0. Dialects show a wider gap: 4.55% macro-average CER on public benchmarks, and 10.38% on the internal 16-dialect test set, against 20.23% for Doubao and 17.13% for Tencent Hy-ASR.

On speaker-diarized transcription, the two variants collectively post the lowest rates across all 12 dataset-metric combinations (DER/WER/cpWER on four test sets); Next reaches DER 2.70 on AISHELL-4, while the cascaded Fun-ASR baseline loses on all four sets. Streaming first-character latency is 100 ms versus 570 ms for Doubao Streaming ASR 2.0 — measured, note, on a self-hosted deployment. On audio understanding, MMAU 81.60% edges past Gemini-3.1-Pro's 80.90%, it sweeps all five specialized tasks (acoustic scenes, non-speech vocal sounds, emotion, musical instruments, speaker tasks), and Clotho-Moment temporal grounding hits 82.20 mIoU, 10.41 points ahead of the runner-up; but Gemini-3.1-Pro keeps the lead on MMSU at 85.30%.

A single-source report; read it rationally

Per the community tracker theopenweights, weights are on Hugging Face under Apache 2.0, with parameter counts undisclosed. All benchmark numbers are vendor-reported; independent replications haven't appeared yet, so keep the attribution when citing specifics.

So what: if meeting notes, call-center QA, or interview transcription is your business, this is the first credible sign that a single open-weights model can replace the cascaded pipeline — but check first whether you need streaming, because the streaming profile does not get diarization.

Reference: technical report arXiv:2610.28711; community listing theopenweights.