IBM Granite Speech 5.0: Drop the language model, 470M parameters transcribe 3.5 hours per second

Event: IBM Pushes ASR Back to Encoder-Only

On August 25, 2026, IBM posted Granite Speech 5.0 TurboCTC on the Hugging Face community blog and released two checkpoints side by side: granite-speech-5.0-470m-turboctc (Apache 2.0) and granite-speech-5.0-470m-turboctc-nc (CC-BY-NC-SA-4.0). Both are 470M parameters and differ only in training data and license; the core architecture is identical (source: IBM Granite HF blog).

The counterintuitive part is that IBM explicitly dropped the language model and LoRA adapters that previous Granite Speech models carried. Earlier generations used a three-stage stack — acoustic encoder + projector + Granite LM + LoRA adapters. The new models revert to a pure encoder + CTC. IBM states this directly: "The encoder-only design provides strong transcription performance, a small memory footprint of only 470M parameters, and over 20× faster throughput than previous Granite Speech models."

The cost of dropping the LM is explicit: the new models give up speech translation and keyword biasing, the two capabilities that benefited most from having an LM in the loop. The benefit lands on the OpenASR Leaderboard: on public English short-form test sets, the non-commercial model scores an aggregate 4.85% WER and the Apache 2.0 model scores 5.00%, with aggregate throughput in excess of 12,600 RTFx. Accuracy and speed arrive in the same package rather than trading off against each other.

Where the Speed Comes From: 50 chars/sec to 12.5 tokens/sec

Granite 5.0 shares four components with its predecessors: a stack of 16 Conformer blocks, self-conditioning after the 8th block, chunkwise attention to avoid quadratic scaling in sequence length, and CTC loss for training. The change concentrates at the output token rate.

Previous Granite Speech encoders emitted 50 characters per second. The new models drop to 12.5 tokens per second, with tokenizers trained on speech transcripts — SentencePiece for the NC model, BPE for the Apache model. Going from the 100-frames-per-second log Mel spectrogram down to 12.5 tokens per second relies on three stages of 2× subsampling. The first stage is a reshape() operation that stacks successive log Mel feature vectors (used by previous encoders too); the second and third stages are embedded in the first two Conformer blocks of the stack and use strided convolutions with stride=2 for temporal downsampling. The convolutional residual path is temporally subsampled by taking the mean of successive pairs of positions so the residual still aligns with the downsampled time steps.

In other words, the speedup is not a clever new algorithm; it is a sparser output. Cutting the token rate to a quarter, combined with dropping the LM that bottlenecked decode, is what lands 12,600+ RTFx on an H200. By IBM's own framing, that is enough to transcribe 3.5 hours of speech in one second under batched inference. On the FFASR far-field leaderboard, the NC model ranks fifth in accuracy and the Apache model ranks ninth — and both are the two fastest models on that board at the same time.

Why This Matters More for Edge Devices

Granite Speech 5.0's positioning is not "beats Whisper" but "ideal for speech-to-text tasks on edge devices." Drop the LM and the model is just an encoder; memory footprint lands at 470M parameters; CTC decoding is greedy with no beam search and no language-model-side KV cache to manage. Hugging Face also hosts a WebGPU streaming demo that runs in Chrome or Edge, which means this 470M-parameter STT can run inside the browser.

IBM also publishes the full training-data recipe. Both models train on seven public datasets (MLS 44,600 hours, YODAS 8,900, CommonVoice-17 2,500, Librispeech 960, VoxPopuli 500, AMI 150, Earnings-22 100). The NC version additionally pulls in GigaSpeech (10,000 hours) and SPGI Speech (4,900 hours). Three synthetic datasets go on top: 2,000 hours of multi-speaker concatenation from MLS/YODAS/CommonVoice/VoxPopuli/AMI, 500 hours of Earnings-22 concatenation, and 240 hours of utterances containing numbers, currencies, website names, phone numbers, addresses, and decimal points — generated by gpt-oss-120b or gpt-oss-20b and synthesized with StyleTTS2. That last block shows, in concrete terms, how a large general model was used to manufacture training data for a much smaller specialist model.

Reader Takeaway: Not Just Another ASR Drop

Read against the second half of 2026, the signal is clear: general large models do not fit every task, and small specialist encoders are pulling ahead on narrow problems. Whisper, Canary, SenseVoice and similar end-to-end STT systems have kept scaling parameters and adding LM-side capabilities; IBM goes the other way and removes the LM. The price is losing translation and keyword biasing. The upside is roughly 1/20 the latency and a fraction of the memory.

For anyone building real-time transcription on terminals, browsers, embedded devices, in-car stacks, meeting minutes, or phone-call analytics where latency matters and translation does not, the Apache 2.0 checkpoint is a near-drop-in production backbone: 470M fits on CPU, CTC decode has no external dependencies, and the 12.5 token/s rate keeps streaming output from falling behind the speaker. The WebGPU demo is the receipt — IBM is showing, not telling, what "drop the LM" buys you in the browser.

If you need speech translation or keyword biasing, the previous Granite Speech models remain IBM's recommended path. The two coexist in IBM's product matrix; they are not substitutes. For everyone else, Granite Speech 5.0 TurboCTC is the one to benchmark first.

References: Granite Speech 5.0 TurboCTC model card (Apache 2.0), WebGPU streaming demo.