FireRedTTS3 README

The project's README from github.com/FireRedTeam/FireRedTTS3. As one markdown file: README.md.

FireRedTTS3

Official PyTorch code for
FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations

FireRedTTS Logo

technical report version HF-model ModelScope-model Apache-2.0

Overview

https://github.com/user-attachments/assets/1ff882ce-828d-48d5-bd2a-4ad12ee912ac

FireRedTTS3 is a unified speech generation and editing system built on semantically enriched continuous speech representations. It comes in two variants:

Highlights ✨

News

Roadmap

Contents

Quick Start 🚀

Installation with pip

pip install -r requirements.txt

Model Download

Download the pretrained model from Hugging Face with the hf CLI:

pip install "huggingface_hub[cli]"
hf download FireRedTeam/FireRedTTS3 --local-dir pretrained_models/

Alternatively, you can download model from ModelScope

pip install modelscope
modelscope download --model FireRedTeam/FireRedTTS3 --local_dir pretrained_models/

Configure Text Frontend

Language Recognition (Optional)

FireRedTTS3-Base relies on explicit language tags for best performance. However, if you don't know the exact language of the text, you can download Meta's FastText language-id model and let it detect the language automatically.

# Download FastText language-id model (lid.176) with:
curl -L -o fireredtts3/utils/llm_tn/models/lid.176.ftz https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz

Text Normalization (TN)

TN converts written numbers, dates, units, currencies, acronyms, etc. into their spoken form (e.g. 19:30 → nineteen thirty). By default, FireRedTTS3 uses the wetext TN tool, which supports Chinese and English, other languages (e.g. Japanese, Russian) undergo only basic cleaning. For full language TN support, enable the LLM-based TN by passing use_llm_tn=True when initializing FireRedTTS3. It reads its config from a .env file:

cp .env.example .env

# Then fill in your values
LLM_TN_API_URL=https://api.deepseek.com/chat/completions   # any OpenAI-compatible endpoint
LLM_TN_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
LLM_TN_MODEL=deepseek-v4-flash                             # or any model >= 30B

Python API

For the best voice cloning performance, use a prompt in the desired language or dialect, since the output inherits the speaking style of the reference. For example, provide a Japanese prompt when synthesizing Japanese and a Sichuanese prompt when synthesizing Sichuanese.

import torch
import torchaudio
from fireredtts3.core import FireRedTTS3

# Init model: choose the text-normalization frontend here.
#   use_wetext=True  -> local weText TN (zh/en only)
#   use_llm_tn=True  -> LLM-based TN (all languages, needs .env / API creds)
#   both False       -> no TN frontend built
tts = FireRedTTS3(
    "pretrained_models",
    use_wetext=True,
    use_llm_tn=False,
)

language = None     # Automatic detection if pass None
prompt_text = "<prompt audio text>"
prompt_audio, prompt_audio_sr = torchaudio.load('prompt.wav')
text = "今天天气很好，我们一起去公园散步吧。"

gen_audio, gen_audio_sr = tts.generate(
    language=language,
    prompt_text=prompt_text,
    prompt_audio=prompt_audio,
    prompt_audio_sr=prompt_audio_sr,
    text=text,
    do_tn=True,       # whether to run the frontend TN on this call
)
torchaudio.save("gen.wav", gen_audio.cpu(), gen_audio_sr)

# Supported languages and dialects

# Multilingual languages:
# Arabic, Cantonese, Chinese, Czech, Dutch, English, Finnish,
# French, German, Greek, Hindi, Indonesian, Italian, Japanese,
# Korean, Polish, Portuguese, Romanian, Russian, Spanish, Thai,
# Turkish, Ukrainian, Vietnamese

# Multi-dialect:
# ZH_Anhui, ZH_Fujian, ZH_Gansu, ZH_Guizhou, ZH_Hebei, ZH_Henan,
# ZH_Hubei, ZH_Hunan, ZH_Jiangxi, ZH_Liaoning, ZH_Minnan, ZH_Ningxia,
# ZH_Shaanxi, ZH_Shandong, ZH_Shanghai, ZH_Shanxi, ZH_Sichuan,
# ZH_Tianjin, ZH_Wenzhou, ZH_Wu, ZH_Yunnan

Instruct API — Voice Design & Speech Editing

FireRedTTS3-Instruct is a unified instruction-driven model. On top of zero-shot voice cloning, it also supports Voice Design, Semantic Edit and Acoustic Edit through a single entry point: fireredtts3.core.FireRedTTS3Instruct.

import torch
import torchaudio
from fireredtts3.core import FireRedTTS3Instruct

# Init the Instruct model (same text-frontend options as FireRedTTS3)
instruct = FireRedTTS3Instruct(
    "pretrained_models",
    use_wetext=True,
    use_llm_tn=False,   # set True to enable LLM-based TN (all languages)
)

# ---- 1) Voice Design Inference ---------------
# Generate a brand-new voice from a natural-language description only;
# no reference audio is needed. The model first writes a voice-attribute
# plan (returned as gen_text), then renders the audio.
instruction = "一个年轻女性的温柔嗓音，语速稍慢，带一点俏皮。"
text = "今天天气很好，我们一起去公园散步吧。"
gen_audio, gen_audio_sr, gen_text = instruct.generate_voice_design(
    instruction=instruction,
    text=text,
)
torchaudio.save("design.wav", gen_audio.cpu(), gen_audio_sr)
print("Voice plan:", gen_text)

# ---- 2) Semantic Edit ------------------------
# Content-level editing: insertion / deletion / substitution by instruction.
# Returns the edited audio and the model's rewritten text with edit mask.
audio_in, audio_in_sr = torchaudio.load("input.wav")
gen_audio, gen_audio_sr, gen_text = instruct.generate_semantic_edit(
    instruction="Replace 'cats' with 'dogs'.",
    audio_in=audio_in,
    audio_in_sr=audio_in_sr,
)
torchaudio.save("edit_semantic.wav", gen_audio.cpu(), gen_audio_sr)
print("Edited text:", gen_text)

# ---- 3) Acoustic Edit ------------------------
# Acoustic-attribute editing: speed / pitch / volume. The instruction must
# follow the trained templates below (free-form phrasing is not supported):
#   speed   ->  "adjust the speed to X"       X in [0.5, 2.0], step 0.1
#   pitch   ->  "shift the pitch by N step(s)"  N in {-6,...,-1,1,...,+6}
#   volume  ->  "adjust the volume to X"      X in [0.3, 2.0], step 0.1
gen_audio, gen_audio_sr = instruct.generate_acoustic_edit(
    instruction="adjust the speed to 0.5x",
    audio_in=audio_in,
    audio_in_sr=audio_in_sr,
)
torchaudio.save("edit_acoustic.wav", gen_audio.cpu(), gen_audio_sr)

# ---- 4) ICL zero-shot voice cloning using the Instruct model ----
gen_audio, gen_audio_sr = instruct.generate_tts(
    prompt_text="<prompt audio text>",
    prompt_audio=prompt_audio,
    prompt_audio_sr=prompt_audio_sr,
    text="<text to be synthesized>",
)
torchaudio.save("gen_instruct.wav", gen_audio.cpu(), gen_audio_sr)

Gradio Demo

# Install gradio
pip install gradio

# Run FireRedTTS3-Base:
python gradio_base.py --host "0.0.0.0" --port 7860

# Run FireRedTTS3-Instruct:
python gradio_instruct.py --host "0.0.0.0" --port 7860

# The Gradio app will be available at http://0.0.0.0:7860.
# If you are serving it behind a proxy, such as Remote VS Code, add the `--root-path` option, for example: `--root-path https://somewebsite/proxy/7860/`

Gradio interface for FireRedTTS3-Base(left) and Instruct(right). Gradio

Performance

Zero-Shot Voice Cloning — Seed-TTS-eval

Best in bold, second best in underline. Evaluation scripts: Seed-TTS-eval.

Model Test-EN
WER/SIM		 Test-ZH
CER/SIM		 Test-Hard
CER/SIM		 Avg
WER/SIM
CosyVoice3-1.5B 2.22 / 72.0 1.12 / 78.1 5.83 / 75.8 3.06 / 75.3
DiTAR 1.69 / 73.5 1.02 / 75.3 – / – – / –
F5-TTS 2.00 / 67.0 1.53 / 76.0 8.67 / 71.3 4.10 / 71.4
FireRedTTS-2 1.95 / 66.5 1.14 / 73.6 8.98 / 70.3 4.02 / 70.1
IndexTTS2 2.23 / 70.6 1.03 / 76.5 7.12 / 75.5 3.46 / 74.2
MegaTTS3 2.79 / 77.1 1.52 / 79.0 – / – – / –
MiniMax-Speech 1.65 / 69.2 0.83 / 78.3 – / – – / –
Qwen3-TTS 1.23 / 71.7 1.22 / 77.0 6.76 / 74.8 3.07 / 74.5
Seed-TTS 2.25 / 76.2 1.12 / 79.6 7.59 / 77.6 3.65 / 77.8
VibeVoice 3.04 / 68.9 1.16 / 74.4 – / – – / –
VoxCPM2 1.84 / 75.3 0.97 / 79.5 8.13 / 75.3 3.65 / 76.7
dots.tts (Pretrain) 1.80 / 77.0 0.97 / 80.4 6.65 / 78.8 3.14 / 78.7
FireRedTTS3-Base 1.64 / 77.2 1.01 / 80.9 6.50 / 78.4 3.04 / 78.8

Multilingual Zero-Shot Cloning — MiniMax-MLS-Test

Best in bold, second best in underline. CER reported for Chinese, Cantonese, Japanese, Korean, Arabic, Vietnamese, Hindi, Thai, and Greek; WER for the rest.

WER / CER (↓) (click to expand)
Language Minimax ElevenLabs VoxCPM2 FishAudio S2 dots.tts (Pretrain) FireRedTTS3
Arabic 1.67 1.67 13.05 3.50 37.91 1.75
Cantonese 34.11 51.51 38.58 30.67 37.91 40.32
Chinese 2.25 16.03 1.14 0.73 1.08 0.91
Czech 3.88 2.11 24.13 2.84 5.05 3.17
Dutch 1.14 0.80 0.91 0.99 1.20 1.15
English 2.16 2.34 2.29 1.62 1.06 2.12
Finnish 4.67 2.96 2.63 3.33 3.44 3.10
French 4.10 5.22 4.53 3.05 3.82 5.28
German 1.91 0.57 0.68 0.55 1.03 0.69
Greek 2.02 0.99 2.84 5.74 2.97 1.24
Hindi 6.96 5.83 19.70 14.64 14.32 7.02
Indonesian 1.24 1.06 1.08 1.46 2.71 1.42
Italian 1.54 1.74 1.56 1.27 3.16 2.28
Japanese 3.52 10.65 4.63 2.76 7.16 3.60
Korean 1.75 1.87 1.96 1.18 5.30 2.42
Polish 1.42 0.77 1.14 1.26 2.72 1.22
Portuguese 1.88 1.33 1.94 1.14 1.64 1.79
Romanian 2.88 1.35 21.58 10.74 3.36 1.93
Russian 4.28 3.88 3.63 2.40 3.64 3.28
Spanish 1.03 1.08 1.44 0.91 0.96 1.21
Thai 2.70 73.94 2.96 4.23 7.45 1.87
Turkish 1.52 0.70 0.82 0.87 5.45 0.92
Ukrainian 1.08 1.00 6.32 2.30 1.61 0.55
Vietnamese 0.88 73.42 3.31 7.41 3.85 0.86
Average 3.77 10.95 6.79 4.40 6.60 3.75
SIM (↑) (click to expand)
Language Minimax ElevenLabs VoxCPM2 FishAudio S2 dots.tts (Pretrain) FireRedTTS3
Arabic 73.6 70.6 79.1 75.0 77.5 78.9
Cantonese 77.8 67.0 83.5 80.5 84.7 83.9
Chinese 78.0 67.7 82.5 81.6 82.3 84.2
Czech 79.6 68.5 78.3 79.8 83.8 86.1
Dutch 73.8 68.0 80.8 73.0 81.4 84.3
English 75.6 61.3 85.4 79.7 86.9 86.8
Finnish 83.5 75.9 89.0 81.9 88.0 89.9
French 62.8 53.5 73.5 69.8 78.2 81.0
German 73.3 61.4 80.3 76.7 79.5 83.3
Greek 82.6 73.3 86.0 79.5 87.6 89.3
Hindi 81.8 73.0 85.6 82.1 84.5 87.2
Indonesian 72.9 66.0 80.0 76.3 80.8 83.3
Italian 69.9 57.9 78.0 74.7 84.5 83.6
Japanese 77.6 73.8 82.8 79.6 83.1 82.8
Korean 77.6 70.0 83.3 81.7 84.3 86.6
Polish 80.2 72.9 88.4 81.9 87.3 89.8
Portuguese 80.5 71.1 83.7 78.1 83.1 86.3
Romanian 80.9 69.9 79.7 73.3 86.2 86.2
Russian 76.1 67.6 81.1 79.0 83.0 84.7
Spanish 76.2 61.5 83.1 77.6 83.9 86.3
Thai 80.0 58.8 84.0 78.6 83.8 83.3
Turkish 77.9 59.6 87.1 83.5 87.4 86.6
Ukrainian 73.0 64.7 79.8 74.7 80.5 79.8
Vietnamese 74.3 36.9 80.6 74.0 80.7 81.3
Average 76.6 65.5 82.3 78.0 83.5 84.8

Instruct TTS

Since Gemini-2.5-pro-preview is inaccessible, Gemini-2.5-pro is used to score all systems.

Model ZH
APS↑ | DSD↑ | RP↑		 EN
APS↑ | DSD↑ | RP↑
MOSS-VoiceGenerator71.6 | 72.5 | 61.358.8 | 71.8 | 61.6
VoiceSculptor-VD74.6 | 63.5 | 62.0– | – | –
Ming-Omni-TTS-16B-A3B84.6 | 70.7 | 56.0– | – | –
Qwen3-TTS-VD83.7 | 81.7 | 65.876.4 | 81.4 | 64.2
FireRedTTS3-Instruct85.8 | 82.0 | 69.780.7 | 82.3 | 72.0

Speech Editing

Semantic Editing (click to expand)
Task Setting Metric Ming-UniAudio-Edit
zh | en		 FireRedTTS3-Instruct
zh | en
Deletion basic WER (%)↓ 11.89 | 14.85 10.51 | 14.46
SIM↑

FireRedTTS3 (FireRedTeam/FireRedTTS3)

Searching Database…

1.8K ·GitHub
FireRedTTS3
这是什么？FireRedTTS3 是一个统一的语音生成与编辑模型：它可以跨 24 种语言和 21 种汉语方言克隆声音，能够根据文字描述设计全新的声音，还能用自然语言指令编辑已有的语音（语义和声学层面）。
这是现成的应用，还是我自己运行的代码库？它是一个 Python 研究代码库，模型权重托管在 Hugging Face/ModelScope 上，你通过 Python API（FireRedTTS3 或 FireRedTTS3Instruct 类）调用它；它还附带一个 Gradio 演示，可以在浏览器中启动进行快速测试。
我怎么让它跑起来？用 pip 安装软件包，从 Hugging Face（使用 hf CLI）或 ModelScope 下载预训练检查点，然后直接调用 Python API 或启动内置的 Gradio 演示。
除 Python 和模型权重外还需要什么？你需要 PyTorch 和已下载的检查点；文本规范化默认使用 wetext 工具（仅支持中文和英文），对于其他语言，你可以选择通过 .env 文件启用 Meta 的 FastText 语言识别模型或基于 LLM 的 TN。
文本规范化在每种语言中效果都一样好吗？不是，开箱即用的完整文本规范化只对中文和英文可靠；日语或俄语等其他语言只能得到基础清洗，除非你开启可选的基于 LLM 的 TN。
FireRedTTS3-Base 和 FireRedTTS3-Instruct 有什么区别？Base 负责跨 24 种语言和 21 种汉语方言的零样本声音克隆，而 Instruct 在克隆的基础上增加了自然语言声音设计（根据描述创造新声音）以及语义/声学语音编辑。
它能克隆说汉语方言的声音吗？可以，只要参考提示使用该方言，FireRedTTS3-Base 就支持跨 21 个汉语方言组（如粤语、四川话和上海话）的零样本克隆。
与其他 TTS 系统相比，它的声音克隆效果如何？在 Seed-TTS-eval 基准上，FireRedTTS3-Base 取得 3.04% 的平均 WER/CER 和 78.8% 的说话人相似度，作者报告其在与 CosyVoice3、F5-TTS 和 Seed-TTS 等系统的对比中名列前茅。
它能做哪类语音编辑？它支持语义编辑（插入、删除或替换词语）和声学编辑（调整语速、音高或音量），两者都由自由格式的自然语言指令驱动。
What are similar projects?OpenVoice, VoxCPM, OmniVoice, AuK, MOSS-TTS-Nano, Index-Translate, FireRedAudio, bark
Which topics is it in?zero-shot-tts, speech, speech-synthesis, voice-cloning, gradio, text-to-speech, multilingual, audio, artificial-intelligence, generative-ai, huggingface, pytorch
Creator
FireRedTeam
Size
0.8 MB
Stars
1.8K
Updated
CommitSep 08
Age
2mo
Commits · 90 Days