Overlapping speakers, mid-sentence code-switching, noisy entertainment audio - the hardest cases for speech foundation models are exactly the ones public corpora lack. Sony's Walrus Technical Report takes a contrarian route: change nothing in the architecture, and spend all the design effort on data.
A borrowed Gemma 4 backbone, an extended speech vocabulary
Walrus is a unified speech-and-audio foundation model with about 8B total parameters (roughly 4B effective, embeddings included), initialized from Gemma 4 E4B with the backbone architecture unchanged; the speech input keeps E4B's USM encoder, unfrozen during fine-tuning. A single set of weights serves every task, selected through natural-language prompts: ASR, code-switching ASR, multi-talker ASR, machine translation, speech translation, speech-to-speech translation, TTS, emotion recognition, audio captioning, and general audio understanding - more than 20 task variants across 100 languages. On the generation side the vocabulary gains 8,192 discrete speech tokens, predicted at 25 tokens per second and 13 bits each, a bit-rate of 325 bps; a single-layer VQ tokenizer plus a GAN vocoder turns them back into waveforms, decoupling token prediction from waveform synthesis.
The real contribution: a three-stage data curriculum with two weights
Training consumed over 600M samples. Stage 1 lays a broad multilingual foundation with recognition-priority task weights, datasets sampled at natural size. Stage 2 starts from the Stage-1 checkpoint and adds timestamp prediction, diarization, synthetic code-switching ASR and dialogue-context tasks, while introducing dataset weights that down-sample older corpora. Stage 3 closes specific gaps: synthetic entertainment speech, low-resource languages, and synthetic multi-talker mixtures of two to four speakers covering Chinese, English, French, German, Italian, Japanese and Spanish. Stage 3 also rebalances task weights - ASR drops from 5.0 to 4.0, TTS rises from 0.25 to 2.0, speech-to-speech translation from 0.5 to 2.0. The recipe for hard cases is blunt: code-switching ASR data is fully synthetic; multi-talker data combines real and simulated recordings; entertainment speech augments permissively licensed collections such as YODAS-Granary; and even the synthetic multi-turn dialogues are generated by three model pairings of Gemma 4 26B and Qwen3.5 9B, then scored 0 to 1 by a Gemma 4 26B judge.
Results: the 4x larger Qwen3-Omni does not pull away
Under the paper's evaluation, Walrus reaches 94.7% word accuracy on FLEURS, ahead of Qwen3-Omni-30B at 93.8% and Whisper Large-v3 at 92.0%. On CS-FLEURS its character error rate is 7.23 versus 32.30 for Gemma 4. On LibriMix clean 2-mix it reaches 19.32 cpWER - half of Qwen3-Omni-30B's 37.04 and far below the dedicated VibeVoice-ASR at 59.34; comparable open SpeechLLMs produce no valid output at all in this setting. On audio understanding, MMAU-Pro reaches 71.92%, above Gemini-Flash-3.7 at 70.40% and GPT-4o-Audio at 52.50%; MMSU reaches 64.1% versus 61.4% for Audio-Flamingo-3; CVSS-C speech-to-speech translation scores 43.63 ASR-BLEU versus 41.27 for the 30B Qwen3-Omni. It does not win everywhere: FLEURS bidirectional speech translation sits at 32.28/22.94, below Qwen3-Omni's 33.60/25.67 - the paper concedes this one, claiming only better speaker similarity and prosody preservation.
The commercial variant's price is written in the licenses
Walrus ships in two variants: Commercial-Ready (CR), trained only on commercially usable data and frozen at the Stage-2 checkpoint, and Research (R), trained on the full curriculum. Moving from CR to R lowers code-switching CER by roughly 19 points, internal entertainment ASR WER by about 9 points across 6 languages, and lifts MMAU-Pro by more than 20 points. In other words, the license terms literally price the capability gap - the most industry-relevant set of numbers in the whole report.
So what
In an era where backbones are becoming public infrastructure, this report turns "data engineering is a primary driver of quality in multi-task speech and audio modeling" into a controlled experiment: fix the architecture, attribute every gain to data. Two lessons for speech teams - the hardest tail cases (overlap, code-switching, entertainment audio) are solved with synthetic pipelines, not by waiting for public corpora; and the line between commercial and research models will increasingly be drawn by which data you are allowed to use, not by architecture. Paper: https://arxiv.org/abs/2611.13689