One H200 GPU, one second, and more than 3.5 hours of English speech transcribed. That is not aggregate cluster throughput — it is the official number from a single 470M-parameter model, IBM's newly open-sourced Granite Speech 5.0 Turbo CTC on Hugging Face, running batched inference. In a year when everyone keeps bolting LLMs onto speech models, IBM went the opposite way: it removed the language model entirely. (Official release blog)

Speed and accuracy, both this time

Per the official figures on the public English short-form test sets of the OpenASR Leaderboard, the two variants — the non-commercial one (4.85% aggregate WER) and the Apache 2.0 one (5.00%) — both deliver aggregate throughput in excess of 12,600 RTFx. For context, the blog states this is over 20x faster than previous Granite Speech models. On the far-field FFASR leaderboard, the two variants rank 5th and 9th in accuracy as of August 25 while also being the two fastest models on that board.

One caveat worth keeping: these are IBM's own official submissions, not third-party reproductions.

Removing the LM is the actual story

The previous Granite Speech architecture was a three-stage pipeline: an acoustic encoder, a projector, and a Granite language model with LoRA adapters. This generation collapses to encoder-only — a stack of 16 Conformer blocks, self-conditioning at the output of the 8th block, chunkwise attention to avoid quadratic scaling with sequence length, and classic CTC loss as the training objective.

The trade-off is stated plainly: capabilities unique to the LM-equipped models, such as speech translation and keyword biasing, are gone. What you get instead is a 470M memory footprint and a deployment profile suited to edge devices. IBM also ships a WebGPU demo of streaming recognition that runs in the browser (Chrome or Edge only).

Low token rate is the real key

The most interesting architectural change is the output token rate: previous Granite encoders emitted 50 characters per second, while Granite 5.0 generates 12.5 tokens per second. Getting from a 100fps log-Mel spectrogram front end down to 12.5 tokens/s relies on three stages of 2x subsampling — the first stacks adjacent feature vectors via a reshape, while the second and third are built into the first two Conformer blocks using strided temporal convolutions. The tokenizers also split: the non-commercial model uses SentencePiece, the Apache 2.0 model uses BPE.

Sequence length drives attention and decoding costs. Cut the token rate to a quarter, stack chunkwise attention on top, and a 20x throughput jump stops being magic and becomes arithmetic.

The data recipe

Both variants share roughly 58,000 hours of natural corpora: MLS (44,600 hours), YODAS (8,900), CommonVoice-17 (2,500), Librispeech (960), VoxPopuli (500), AMI (150), and Earnings-22 (100). The non-commercial variant adds GigaSpeech (10,000 hours) and SPGI Speech (4,900) — which is exactly why its WER is slightly lower.

Synthetic data fills three gaps: 2,000 hours of multi-speaker concatenations from MLS/YODAS/CommonVoice/VoxPopuli/AMI, 500 hours of meeting-style concatenations from Earnings-22, and 240 hours targeting traditional ASR weak spots — numbers, currencies, website names, phone numbers, and addresses — with text generated by gpt-oss-120b or gpt-oss-20b and synthesized into speech via StyleTTS2. Using open-weight LLMs to manufacture ASR training data is a loop worth pondering on its own.

So what

The dominant speech-recognition narrative of the past two years has been "bigger is better": hang an ever-larger language model behind the encoder and let LLM world knowledge do the patching. Granite Speech 5.0 Turbo CTC offers evidence in the other direction: if your use case is transcription itself, a 470M encoder-only model with a low token rate can open up an order-of-magnitude throughput lead while keeping a commercially friendly Apache 2.0 license.

The variant choice is laid out cleanly: take the non-commercial build for maximum accuracy (CC-BY-NC-SA, no commercial use), or take Apache 2.0 for deployment and pay 0.15 WER points extra. For teams building edge devices, live captions, or bulk transcription pipelines, this is an engineering lesson you can copy directly: lower the token rate first, then talk about everything else.