Real-Time-Voice-Cloning README

The project's README from github.com/CorentinJ/Real-Time-Voice-Cloning. As one markdown file: README.md.

Real-Time Voice Cloning

This repository is an implementation of Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (SV2TTS) with a vocoder that works in real-time. This was my master's thesis.

SV2TTS is a deep learning framework in three stages. In the first stage, one creates a digital representation of a voice from a few seconds of audio. In the second and third stages, this representation is used as reference to generate speech given arbitrary text.

Video demonstration (click the picture):

Toolbox demo

Papers implemented

URL Designation Title Implementation source
1806.04558 SV2TTS Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis This repo
1802.08435 WaveRNN (vocoder) Efficient Neural Audio Synthesis fatchord/WaveRNN
1703.10135 Tacotron (synthesizer) Tacotron: Towards End-to-End Speech Synthesis fatchord/WaveRNN
1710.10467 GE2E (encoder) Generalized End-To-End Loss for Speaker Verification This repo

Heads up

Like everything else in Deep Learning, this repo has quickly gotten old. Many SaaS apps (often paying) will give you a better audio quality than this repository will. If you wish for an open-source solution with a high voice quality:

Running the toolbox

Both Windows and Linux are supported.

  1. Install ffmpeg. This is necessary for reading audio files. Check if it's installed by running in a command line
ffmpeg
  1. Install uv for python package management
# On Windows:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
# On Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Alternatively, on any platform if you have pip installed you can do
pip install -U uv
  1. Run one of the following commands
# Run the toolbox if you have an NVIDIA GPU
uv run --extra cuda demo_toolbox.py
# Use this if you don't
uv run --extra cpu demo_toolbox.py

# Run in command line if you don't want the GUI
uv run --extra cuda demo_cli.py
uv run --extra cpu demo_cli.py

Uv will automatically create a .venv directory for you with an appropriate python environment. Open an issue if this fails for you

(Optional) Download Pretrained Models

Pretrained models are now downloaded automatically. If this doesn't work for you, you can manually download them from Hugging Face.

(Optional) Download Datasets

For playing with the toolbox alone, I only recommend downloading LibriSpeech/train-clean-100. Extract the contents as <datasets_root>/LibriSpeech/train-clean-100 where <datasets_root> is a directory of your choosing. Other datasets are supported in the toolbox, see here. You're free not to download any dataset, but then you will need your own data as audio files or you will have to record it with the toolbox.

Real-Time-Voice-Cloning (CorentinJ/Real-Time-Voice-Cloning)

Searching Database…

60.2K ·GitHub
Real-Time-Voice-Cloning
这是什么？这是 SV2TTS 的一个实现。SV2TTS 是一种深度学习框架，只需几秒钟的音频就能克隆一个人的声音，然后实时生成这个声音的任意语音。
这是打包好的应用，还是研究用的工具箱？这是一个研究用的工具箱（最初是作者的硕士论文），由三个相互独立的深度学习阶段组成：说话人编码器、Tacotron 合成器和 WaveRNN 声码器。它通过命令行运行，而不是作为独立应用安装。
我该怎么让这个工具箱运行起来？先安装 ffmpeg（用于读取音频文件），再安装 uv（用于管理 Python 包），然后运行提供的某个命令。uv 会自动创建一个包含正确 Python 环境的 .venv，预训练模型也会自动下载。
运行之前，我需要先安装什么？需要 ffmpeg 来读取音频文件，需要 uv 来管理 Python 包。预训练模型会自动获取；如果自动获取失败，可以手动从 Hugging Face 下载作为备选。
在语音克隆方面，它现在还是最先进的吗？不是。作者表示这个仓库过时得很快，现在许多付费 SaaS 应用的音频质量更好。他推荐“Chatterbox”作为更新的开源替代方案。
它能同时在 Windows 和 Linux 上运行吗？可以，Windows 和 Linux 都被明确支持用于运行这个工具箱。
它支持哪些数据集？如果只想单独体验这个工具箱，作者推荐下载 LibriSpeech/train-clean-100 并解压到本地。也支持其他数据集，你也可以使用自己的音频，或者直接在工具箱中录音。
作者为什么建议先看看其他项目？因为深度学习研究发展很快，这个仓库现在已经过时。他建议在 Papers with Code 上浏览较新的语音合成研究，或者试试“Chatterbox”，它的语音克隆质量达到了 2025 年的水平。
它实现了哪些论文？它实现了 SV2TTS（说话人验证到语音合成的迁移学习）以及 GE2E 说话人编码器损失函数本身，并在声码器和合成器阶段复用了 fatchord/WaveRNN 中的 WaveRNN 和 Tacotron 实现。
What are similar projects?VoxCPM, TTS, MockingBird, FireRedTTS3, OpenVoice, MOSS-TTS-Nano, bark, FireRedAudio
Which topics is it in?ai-research, speech, voice-cloning, speech-synthesis, tensorflow, tts, text-to-speech, neural-network, real-time, audio, huggingface, pytorch
Creator
CorentinJ
Size
361.1 MB
Stars
60.2K
Updated
CommitMar 09
Age
7y
Commits · 90 Days