In a traditional speech pipeline, transcribing what was said and figuring out who said it have always been two separate jobs: an ASR model handles the former, a speaker diarization system handles the latter, and only by stitching the two together do you get meeting minutes with names attached. Offline, that split is fine — the audio is finished anyway. Real-time voice assistants and agents cannot afford it: waiting for the full audio before running diarization blows the latency budget. A technical report submitted to arXiv on September 2 by Microsoft Research offers a new answer: VibeVoice-ASR-Streaming, an LLM-based end-to-end streaming speaker-attributed transcription model (arXiv:2609.02812).
Streaming who-said-what as speech arrives
The core mechanism interleaves fixed-size audio chunks, a small amount of lookahead audio, and previously generated text, so the model keeps producing who said what as speech arrives — with no separate diarization stage. The paper describes this as one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. Keep the word “describes” in mind: this first-of-its-kind claim currently rests on a single source, and by usual practice deserves full credit only after community reproduction.
The evaluation numbers also come from the paper’s self-reported setup: the 7B model achieves the lowest average WER/CER across five evaluation sets, and best or tied-best speaker attribution on 12 of 13 evaluation settings. Alongside the 7B model there is a 1.5B variant; both weight sets ship with inference code, the GitHub repository is microsoft/VibeVoice, and the Hugging Face model card lists an MIT license.
Two engineering details stand out for practitioners. First, customized hotwords: names and technical term lists can be injected to improve recognition of domain-specific content. Second, ten-language support — Chinese, English, French, German, Italian, Japanese, Korean, Portuguese, Russian, and Spanish — enough coverage for mainstream meeting scenarios.
Why voice agents need this
The real weight of this work sits in the latency budget. In multi-turn real-time agent conversations, if speaker identity only arrives after a diarization post-process, the context breaks: the agent cannot tell whether the incoming utterance came from the user or a third person in the background. Baking attribution into streaming decoding itself is what lets agents in multi-speaker scenes know on the spot who is talking — a key step for voice interaction moving from one-speaker, one-listener setups into real multi-person environments. The 1.5B release also gives low-resource deployments an explicit option, since not every edge device can fit 7B.
In product-line terms, the VibeVoice family was previously known for long-form multi-speaker TTS; this release fills the ASR side. Microsoft is betting open weights on both the generation and understanding ends of voice — a clear ecosystem play for voice-agent infrastructure.
So what
For speech product teams, the value here is not the benchmark scores but the bundle: streaming, speaker attribution, and hotwords in one model, under an MIT license. Self-hosted real-time meeting transcription and multi-speaker voice-agent memory now have weights to grab directly. The one caveat: both the evaluations and the first-of-its-kind framing are vendor-reported — run your own eval set before making procurement decisions.