LoRA modules that refine phoneme-level pronunciation and prosody on top of LLM-based TTS with no explicit G2P frontend. Lightweight, easy to drop in, research-friendly.
We currently support Japanese on CosyVoice 2 only.
Add controllability to LLM-TTS with no explicit G2P frontend. Objective and subjective evaluations confirm its effectiveness: Accent correctness has been improved from 0.472 (baseline) to 0.975 (UtterTune) while keeping naturalness and speaker similarity.
UtterTune tunes only the target language. You don't need attach UtterTune on other languages, so it never degrade their performance.
Attach to the LLM backbone via PEFT, keeping base knowledge as well as possible.
Tiny adapter weights, fast to download and update. You can train LoRA on a single Nvidia RTX4090 (24GB memory) within 1 hour.
The code is released under MIT License. The currently available LoRA weights are released under CC BY-NC 4.0 considering the licenses of the training data.
Baseline refers to the original CosyVoice 2. Baseline (kana input) refers to the original CosyVoice 2 and kana (phonogram) input for difficult-to-read words.
Input:
Input:
Input:
Synthesize your own text on a free ZeroGPU Space.
Tag words with <PHON_START>/<PHON_END> to control pronunciation.
Only the sample prompt speakers above can be used (no arbitrary reference-audio upload), to keep this public endpoint from being usable for general-purpose voice cloning.
First request after idle may take up to ~1 minute while the Space wakes up / a GPU is allocated.
Example (click to use):
<PHON_START>チ'ミ/モーリョー<PHON_END>が<PHON_START>バ'ッコ<PHON_END>する。
If you use UtterTune in your research, please cite the paper:
@misc{Kato2025UtterTune,
title={UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech},
author={Kato, Shuhei},
year={2025},
howpublished={arXiv:2508.09767 [cs.SD]},
}
The model used in this demo is released under CC BY-NC 4.0. It inherits the non-commercial restriction from the training data. Allowed: Academic research, non-commercial research (including within commercial companies), and personal use. Commercial use is prohibited.
The code used in this demo is released under MIT License.
We attached a PEFT adapter targeting all the Q/K/V/O projections in the LLM portion used by CosyVoice 2.
The existing available model is released under CC BY-NC 4.0 considering the licenses of the training data. You can train your own model using our code, released under MIT license.
Yes. The eval-audio-generation, CER, speaker-similarity, UTMOSv2, and significance-test scripts used to produce Table 1 are published in the GitHub repository, along with the training manifest and evaluation sentence sets.
The live demo is a public, free endpoint, so we intentionally limit it to a fixed set of sample prompt speakers rather than allowing arbitrary voice cloning. If you want to use your own reference audio, run the code locally with scripts/cv2/infer.py, which has no such restriction.