Spoken dialogue models carry a stubborn cost: for every second of speech, the model autoregressively emits a stream of speech tokens, one step at a time. Mainstream SDMs share one speech-token vocabulary between generation and synthesis, and that vocabulary typically runs at 25 Hz — Step-Audio 2 mini's S3Tokenizer V2 runs at 25 Hz with 6,656 speech units, so one second of speech means 25 autoregressive steps. Longer conversations mean costlier inference and harder latency targets.
Swapping in a lower-rate tokenizer is the obvious fix, but it is invasive: compression-quality trade-offs appear, and the original tokenizer–decoder pipeline has to be replaced, which strands the trained model. CoSTA (paper), posted to arXiv on October 15 by a team from the Institute of Automation, Chinese Academy of Sciences, takes a third path: keep the original tokenizer and decoder untouched, and extend only the generation side with a compressed vocabulary — the model learns to "speak short" while the decoder keeps "listening long."
Three post-hoc steps, about 6% more parameters
The method has three stages. First, vocabulary extension: speech-token sequences are treated like text and compressed with BPE; 80K compressed units learned from dialogue speech tokens are added to the model vocabulary. Each new unit's embedding and LM-head row is initialized with the mean of its component tokens' rows; original vocabulary parameters stay frozen, and the extension adds only about 6% of the original parameter count. Second, pronunciation adaptation: 300 hours each of Chinese and English speech from Emilia, timbre-unified with Seed-VC, teach the model to do TTS in the compressed token space. Third, dialogue adaptation: roughly 50K speech-dialogue examples per language, constructed following OpenS2S with assistant speech synthesized by CosyVoice 2 in a consistent voice, update only the LLM parameters while the speech encoder, adaptor, and speech decoder stay frozen. At inference, the compressed tokens the model emits are deterministically expanded back into the original speech-token sequence and fed to the untouched decoder for waveform synthesis — the compression is lossless and the synthesis interface is unchanged.
The efficiency ledger: 1.7x generation, latency down by a third
Under a vLLM streaming inference setup, the paper reports CoSTA-ZH compresses generated tokens to 66.81% with 1.736x relative speed; the English variant reaches 70.17% and 1.659x. The control, Vanilla-FT, fine-tunes on the same adaptation data with original speech tokens: compression stays at 100% and speed barely moves — the gains come from compressed tokens themselves, not from extra training. At the system level, full-response latency drops 32.8–39.0% in Chinese and 22.4–26.0% in English; throughput rises 51.6–59.2% and 25.8–38.6% respectively; peak memory grows by at most 0.68 GiB. On quality, across URO-Bench's Chinese and English datasets, Chinese G-Eval rises from 69.24 (base) and 69.88 (Vanilla-FT) to 72.27, ASR error falls from 7.49% / 4.96% to 1.51%, and UTMOS speech quality holds steady; across 180 paired human ratings per language and metric, win-or-tie rates reach 86.1–96.1% for fluency and 83.3–87.7% for semantic quality.
The ablation that carries the paper
The most informative result is the component ablation: remove dialogue adaptation and G-Eval collapses from 72.27 to 24.35 — learning to "pronounce in compressed form" is not the same as learning to "hold a dialogue in compressed tokens." The vocabulary-size trade-off is equally concrete: in the vocabulary experiments, growing from 80K to 160K added units pushes the compression rate from 67.50% down to 63.62% but buys only 1.1% more speed while G-Eval slips from 81.36 to 78.87, which is why 80K is the paper's chosen balance. The limited-budget stress test is vivid: under the same generation budget, Chinese sustains 7.21 effective turns (Vanilla-FT 3.71, base 2.14) and English 5.79 (1.84). Transferred to OpenS2S — a different architecture with a Think–Talk design and a 12.5-Hz GLM-4-Voice tokenizer — CoSTA still cuts speech-token steps by 21.17% / 18.64% with slightly better quality, showing the post-hoc recipe can stack on top of low-rate tokenizers. The team also reports an approximately log-linear decline of compression rate with vocabulary size across 10K–320K added units and eight configurations.
Where to be skeptical
Every benchmark number is self-reported by the paper and its official GitHub repository; there is no third-party replication yet. The authors themselves note the stress-test turn counts measure "generation-format sustainability rather than conversational coherence," and the efficiency results are tied to the vLLM streaming setting. Worth noting too: the paper sat on arXiv since October 15 and only entered community view when it was submitted to the Hugging Face papers board on October 30 — good work sometimes has to wait for its window.
While the field races on parameters and full-duplex capability, CoSTA points at a plainer lever: make the model emit fewer tokens per second of speech. For latency, the most expensive token is the one you never have to generate.