A voice agent has to transcribe people while they are still talking, and it has to do that across thousands of concurrent calls. Making a model ignore future audio usually means masking attention, but masked attention still caches every past key and value. The paper puts a number on that: an 18-layer, d=1024 encoder at 80ms frames accumulates roughly 110MB of fp32 cache per minute of audio per caller, and an unbounded causal attention encoder of the same size passes 1GB after ten minutes. State that grows with call length means server cost that grows with call length.
E-Gaborformer (arXiv:2611.08092, https://arxiv.org/abs/2611.08092, by Nir Krakowski, Tamir Cohen, Ohad Zohar and Chen Zarfati) tries to solve both constraints at once, by refusing to let attention carry the memory.
Compressing the past into a fixed-size state
E-Gaborformer keeps the E-Branchformer block but changes three things inside it.
First, the global branch. Layers follow a GGGS pattern: three Gated DeltaNet-2 recurrent layers (GDN-2) for every fourth layer of causal sliding-window attention over a 128-frame window, about 10.24 seconds. The recurrent layer updates one fixed-size matrix state per frame, 64x64 in all models, while each attention layer keeps a ring buffer of its most recent 128 frames to provide the exact recall a compressed state cannot.
Second, the convolution kernels. Instead of letting the network learn them freely, the paper locks them into the span of a fixed dictionary of causal Gabor atoms: 16 unit-norm atoms made of two Gaussians and seven log-spaced wave packets, frequencies from 0.03 to 0.4 cycles per frame, each split into cosine and sine. With a kernel of 31 and 80ms frames, one filter spans 2.48 seconds and the wave packets center between 0.375 and 5Hz, the band of phrasal and slower syllabic modulation. Each channel learns 16 coefficients instead of 31 numbers, and causality is automatic because the envelope peaks at the current frame and decays into the past. The dictionary is ill-conditioned, with a Gram condition number around 8x10^6.
The counterintuitive bug: the recurrent branch starts mute
In the first round of 18 causal runs with GDN-2 layers, four never left the initial CTC plateau within an epoch: the model emitted only blanks and the loss stayed flat.
The cause was the output path. The GDN-2 branch output had an RMS of only 0.005 to 0.006, against 0.28 to 0.38 for the attention branch of the same model. The recurrent output fell below the normalization epsilon and RMSNorm could not rescale it, so the network learned to work without the recurrent branch.
The fix is cheap: replace the SiLU output gate with a sigmoid (starting at 0.5) and lower the RMSNorm epsilon from 1e-5 to 1e-6. Neither change adds parameters. The branch then starts at an RMS of about 0.09, and all 36 later causal runs with GDN-2 layers left the plateau, against 4 of 18 before.
That detail is worth keeping: in hybrid architectures the failure mode is often not the architecture, but one branch being quietly killed off by its numerical range.
A controlled comparison against E-Branchformer
The paper runs a strictly causal controlled study on LibriSpeech-100, where every model shares one front end and one recipe and only development sets are reported. The study model has 12 layers at d=256 and 27.6M parameters, against a 26.8M-parameter E-Branchformer ported from ESPnet. After three epochs, E-Gaborformer scores 26.2/44.5 on dev-clean/dev-other versus 27.1/45.1 for the baseline, leading by 1.4, 0.2 and 1.2 points on dev-clean across three seeds; after ten epochs both reach 39.2% on dev-other and E-Gaborformer stays 0.3 points lower on dev-clean.
Reaching a given dev-clean WER comes 5 to 18% sooner than the baseline, across an architecture screen of 36 encoder configurations and roughly 160 training runs. Replacing both Gabor convolutions with unconstrained depthwise kernels costs 1.2 points on dev-clean, and removing all attention layers costs 1.1 and 1.4. The 3:1 hybrid matches the 1:1 hybrid while using half the attention layers.
At production scale: 19MB and the Nemotron comparison
Scaled up, the production model has 18 layers at d=1024, 14 GDN-2 layers against 4 sliding-window attention layers, and a 598M-parameter encoder with an RNN-T decoder, for 626M parameters in total, roughly the size of Nemotron 3.5 ASR Streaming (0.6B), NVIDIA's public 600M-parameter streaming recognizer built on a Cache-Aware FastConformer-RNNT architecture. Training used about 12k hours of Hebrew and 96k hours of English, and during the first 10,750 updates Nemotron served as a teacher.
On the state side: 3.7MB of GDN-2 states, 4.2MB of attention key/value windows, 8.8MB of Gabor convolution caches, plus about 2MB for the prediction network, totaling roughly 19MB regardless of call length.
The comparison comes from real English voice-agent traffic, 24 recordings with 1,214 reference words after normalization. Nemotron 3.5 ASR Streaming scores 14.0 causally and 13.3 with 480ms lookahead, while E-Gaborformer scores 9.6 in both modes. That is 31% lower causally, and 28% lower once Nemotron gets its lookahead. Lookahead helps Nemotron by 0.7 points and costs E-Gaborformer nothing, because only 4 of its 18 layers can see future frames at all.
The paper qualifies that comparison itself: this traffic is what E-Gaborformer was trained on, with Nemotron having served as an early teacher. It is closer to an engineering self-demonstration in one setting than to an independent ranking.
Against the authors' previous production model, Hebrew WER drops 11% to 20% relatively across three of four Hebrew sets (general 12.3 to 10.9, hard 14.2 to 11.5, name-heavy 10.2 to 8.2) while English holds steady. The one regression is the addresses, URLs and emails set, up 7%.
Fixing the RNN-T training math
The paper also treats a long-standing RNN-T training inefficiency: a standard implementation runs its dynamic program over the full-vocabulary log-softmax even though the loss needs only two log-probabilities per lattice node, either emitting the next token or emitting blank.
The fix moves the forward-backward recursion onto a two-class lattice and derives the logit gradient in closed form, leaving loss and gradients identical to the full lattice. Supporting optimizations include padding the output layer to a multiple of 128 rows so the 13,101 outputs return to tensor-core shapes, rewriting depthwise convolutions in Toeplitz form, and repacking the utterances of every micro-batch of one update by length, raising the fill rate from 61% to 80-83%. Per update time falls from 7.55 to 2.90 seconds on one GPU, a 2.6x speedup, and from 9.0 to 3.58 seconds, or 2.5x, across three GPUs with live data.
So what
The value of this paper is not another lower ASR WER. It is that it treats streaming state as a first-class design goal. Attention has grown stronger, and with it the default assumption that state grows with audio length. E-Gaborformer chooses differently: recurrence carries long-range memory, attention supplies exact recall, and a fixed dictionary of convolution kernels shrinks the feature-extraction space, for a state size that does not depend on call length.
For people building agents the trade is direct: in long conversations, state that stops growing means concurrency cost that stops growing. Nineteen megabytes sounds small, but multiply it by concurrency and by call duration and you get a completely different cost curve.
The paper's own next steps are honest: longer controlled training runs, emission-latency measurements, and evaluation on multi-minute streams, since the constant-state design matters most exactly in the scenario it has not yet tested.
Which raises the question: if the recurrent state, the attention window and the convolution history are all fixed, what does the idea of context even mean in a streaming speech model?