Full-duplex speech models have an awkward resource problem: they listen and speak at the same time, and while they listen, most of the GPU sits idle waiting for the next audio chunk. A new paper proposes turning that idle time into useful work.

Paper: arXiv:2611.06982

The silence hypothesis

Chunk-interleaved full-duplex models split a conversation into fixed time chunks, and each chunk must finish before the next one arrives. Profiling four such models chunk by chunk, the authors found the workload is lopsided: listen chunks leave compute idle for 70 to 89 percent of the chunk duration, while speak and tool-call chunks need serial generation that grows with output length. Every tool call they evaluated missed its deadline, and LWS, which decodes a written channel alongside speech, missed the deadline on 95 percent of speak chunks.

The authors then ran a prediction probe: decode every chunk twice, once with the real audio and once with a chunk of silence, and compare the two outputs. Two properties held across all four models. First, the outputs matched token for token in 95.3 percent of chunks, so the next audio chunk usually does not change the output at all. Second, of the chunks that did diverge, 87.7 percent diverged at the first output token, which is where most models decide whether to keep listening or start responding.

Three components

Speculative Listening is training-free and lossless: it adds no parameters and preserves the target output distribution.

First, drafting under the silence hypothesis. During the idle window, the model assumes the next chunk is silence and greedily decodes a draft.

Second, a cross-chunk hypothesis pool. Drafts are indexed by N-grams and kept across chunks, because a user request spans several chunks and drafts made earlier already contain much of the eventual tool call.

Third, a runner-up branch. A second draft starts from the second-highest first token, covering the cases where the primary draft diverges immediately.

Once the real audio arrives, a single forward pass prefills the audio and verifies the whole draft, so post-audio computation grows only with the tokens that actually changed.

What the ablation shows

On Streaming SLM, adding silence-hypothesis drafting alone speeds up speak chunks by 2.6 to 2.9x, but tool calls only by 1.2 to 1.4x: a call with several dozen tokens does not fit in a single idle window. Adding the hypothesis pool lifts tool-call speedup from 1.42x to 22.04x. Adding the runner-up branch pushes it to 23.41x.

So most of the gain does not come from guessing more accurately. It comes from accumulating drafts across chunks.

Numbers and costs

On TickTock-Bench, with 760 sessions and 15,883 chunks across four models (Streaming SLM, Gander, Realtime-Venus, and LWS), speak chunks speed up by 2.6 to 2.8x on average and tool-call chunks by up to 23.41x, versus 3.45x on Gander. The three training-free baselines (PLD, Token Recycling, Lookahead) reach only 0.96 to 1.23x. The decoding delay before a response starts drops 73 percent on average, to 74 to 241 milliseconds, close to the roughly 200 millisecond gap between human turns. Maximum lag behind real time falls by 94.4 percent on average. Response quality holds: AlpacaEval scores differ from autoregressive decoding by less than 0.1 points.

The costs are equally direct. This targets low-concurrency, latency-sensitive deployment. Going from one concurrent session to four, the tool-call speedup drops from 22.7x to 10.5x. The authors also note that they measured only the Thinker and not the Talker, and that parallel-stream architectures that decode every 80 milliseconds are out of scope.

Why it matters

Speculative decoding papers have spent recent years squeezing more speed out of the same model. This one changes the question: instead of squeezing idle compute, it asks what that idle compute should have been doing in the first place. Full-duplex speech is one of the few workloads with large idle windows by construction, since the GPU waits on every millisecond of user silence. Turning waiting for input into a resource looks to me more valuable than any single kernel-level optimization.

The caveat is that the approach only applies to architectures that have an idle window, and to low-concurrency serving. It also leaves an obvious question open: if idle compute is this valuable, why is it not standard in every speech stack? The likely answer is that changing a runtime is harder and riskier than changing a model, and papers can afford to look clever where products cannot.