A single Japanese Kanji tells you how deep the polyphony rabbit hole goes: 「秋」 is aki standalone, shū inside 「晩秋」, and toki in the classical phrase 「危急存亡の秋」. The title of A Certain Scientific Railgun pushes it further — 「超電磁砲」 is canonically chōdenjihō, yet the work demands the English loanword rendering 「レールガン」 (Railgun). Getting text-to-speech to read these correctly is one of the most stubborn problems in Japanese voice synthesis.
End-to-end TTS has made the hole deeper. Cascaded systems first convert text to phonemes through a front-end: pronunciation is controllable, but the semantic and prosodic cues carried by the Kanji glyphs are stripped away. Newer zero-shot models consume raw orthographic text directly — naturalness improves, yet polyphone errors persist. Ruby-OPD, a paper posted to arXiv on November 4 by a Baidu Shenzhen team, quantifies the floor: on the Joyo-Kanji-Yomi benchmark (JKYB), general multilingual models reach only 72.10% reading accuracy for Qwen3-TTS 1.7B, 73.29% for FireRedTTS3, and 89.90% for FishAudio S2 Pro, with target-word Kana error rates climbing to 23.68%.
Welding an annotation channel into training
The first stage tackles controllability. Borrowing the Ruby typography that annotates Kanji with small Kana in Japanese books, the team defines implicit (parenthesized pronunciation) and explicit (boundary-marked) annotation formats. Fine-tuning on annotation data alone fails: pretrained on raw text, the base model can already guess most readings, so it has little incentive to attend to annotations and even reads out both the glyphs and the annotations at inference. The fix is orthography perturbation — during training, randomly mask Kanji spans, substitute unrelated characters, or swap in semantically similar words with entirely different pronunciations (replacing the Kana word 「ワールド」, wārudo, with 「世界」, sekai), forcing the model to rely on the annotation. One counter-intuitive ablation: converting the whole sentence to Kana performs worse than converting only the target word — Kanji glyphs are the anchors for segmentation and meaning, and stripping them sentence-wide guts the model's context.
On-policy distillation without paired speech
The second stage tackles raw-text accuracy. A frozen teacher reads fully annotated text; the student reads raw text. Intermediate states are rolled out online by the student from pure Gaussian noise, carrying zero information leakage from real speech, and the student is trained to match the teacher's velocity field at those states. Seeded from UniDic dictionary entries, Gemini 3.1 Pro synthesizes roughly 680k monophonic and 100k polyphonic sentences — the distillation core needs only (text, annotation) pairs, with paired speech interleaved at just 20% of steps to anchor audio quality. The results are decisive: SFT on annotation data alone drops raw-text accuracy from the base's 94.11% to 92.32%; adding on-policy distillation lifts it to 97.13%, halves target Kana error from 4.68% to 2.02%, and reaches 0.70% sentence-level error — the top accuracy in the paper's eight-model Table 2. The JKYB benchmark itself spans 13,536 sentences, 2,136 regular-use Kanji, and 4,512 officially recognized readings.
Costs and takeaways
The honest part: voice-cloning similarity (CAM++) sits at 0.703, slightly below the base's 0.728, while speech-quality metrics UTMOSv2/DNSMOS stay on par with the teacher. The gains in controllability and accuracy cost a sliver of cloning similarity. The base model, Irodori-TTS-v4.1-Small, is open source, and the 774 hours of aligned SFT audio come from the Japanese subset of the open Emilia dataset — the whole recipe is reproducible in engineering terms.
For Chinese, the recipe is nearly transplantable: Chinese polyphones (the 「重」 in 「重庆」, the 「行」 in 「银行」) share the same root ambiguity as Japanese Kanji readings, and the combination of surface-form preservation, an annotation channel, and perturbation training carries over directly. The broader signal is worth more: pronunciation knowledge can be distilled into a model from plain text alone, without burning another round of studio recordings per language. As speech synthesis competition shifts to long-tail languages, whoever squeezes "text as the textbook" hardest crosses the data bottleneck first. Paper: https://arxiv.org/abs/2611.06403