The standard recipe for voice cloning bolts a separate speaker encoder onto the generator: feed in a few seconds of reference audio, get a fixed vector back, and the generator copies the voice from that vector. An RSE-TTS paper from the VoxNexus team, posted to arXiv on November 11, takes that convention apart: if a pretrained TTS generator has already learned to align text and speech, why not let it serve as its own encoder?

A generator repurposed for compression

The paper starts with a clean controlled study: fix the architecture and training pipeline, and only change how the encoder is initialized. A randomly initialized independent encoder reaches only 0.400 speaker similarity (SIM) on Seed-TTS; copying the pretrained TTS backbone weights in lifts SIM to 0.530, with CV3-eval rising from 0.342 to 0.469 and MiniMax from 0.391 to 0.592. Generative skill transfers into understanding - not as a slogan, but as numbers.

Text adds more. Feeding the reference audio together with its transcript raises SIM on every benchmark - even though what the encoder compresses is sound, knowing what was said helps it remember who said it.

The full system, RSE-TTS, builds on this: a Reconstructive Speaker Encoder shares the generation backbone, passing reference text, reference speech, and eight learned queries through one encoding pass to produce eight continuous conditioning vectors. Sharing eliminates a second Transformer entirely: trainable encoding-plus-generation parameters drop from 228.5M to 118.2M, a 48% reduction.

The transcript is not decoration

One might suspect the transcript acts as a generic hint. The mechanistic experiments say otherwise. Feeding the reference transcript with tokens in reverse order - same tokens, scrambled order - drops SIM from 0.538 to 0.497. Order itself carries information.

Activation patching then traces the route: replacing layer-3 audio hidden states in the reversed-text run with correct-text states restores SIM to 0.545, nearly closing the gap; at layer 9, it is the eight query positions that matter, with replacement recovering SIM from 0.497 to 0.542. The text benefit travels a concrete path: first into intermediate acoustic representations, then into the eight compressed vectors.

Those eight vectors hold more than timbre. Growing the reference from 3 to 24 seconds raises SIM from 0.378 to 0.569, with all 30 test speakers improving at 24 seconds - the compression bottleneck is unchanged, yet it absorbs more. References read fast shorten generated speech by 7.0% on average, slow references lengthen it by 13.4%, and high-pitched references raise output pitch by 6.01 semitones: delivery is encoded alongside identity.

The 118M generator in practice

At full training scale, the generator has 118.2M parameters, about 129M for the whole generation chain including the waveform decoder. Against the official MOSS-TTS-Nano in both its cloning and continuation modes, RSE-TTS wins on word error rate (WER) and SIM across all three English benchmarks - in reference-cloning mode, Seed-TTS WER falls from 3.96% to 1.49% and SIM rises from 0.480 to 0.598.

Stacked with acoustic continuation (keeping reference speech as acoustic history), its WER comes in below the evaluated CosyVoice 3 and MOSS-TTS-Local on every benchmark - generation chains 6.6x and 30.5x larger respectively. SIM still trails those bigger systems, and the paper does not hide it: compact conditioning caps similarity below continuation-style schemes, which the authors suspect comes from teacher forcing encouraging over-reliance on acoustic history. RSE also requires a transcript of the reference audio, so transcription robustness matters for audio-only uses, and the experiments cover English only.

The signal here outlasts the model itself: the text-speech alignment learned inside a pretrained generator is a reusable asset, not a single-purpose tool. For on-device and low-cost voice cloning, folding the encoder into the generator may be a cheaper path than piling on data. If eight vectors can carry timbre, pace, and pitch, the next question is: which other components never needed to exist separately?

The paper and audio samples are at arXiv 2611.15464 and the RSE-TTS demo page.