Real-Time-Voice-Cloning README

The project's README from github.com/CorentinJ/Real-Time-Voice-Cloning. As one markdown file: README.md.

Real-Time Voice Cloning

This repository is an implementation of Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (SV2TTS) with a vocoder that works in real-time. This was my master's thesis.

SV2TTS is a deep learning framework in three stages. In the first stage, one creates a digital representation of a voice from a few seconds of audio. In the second and third stages, this representation is used as reference to generate speech given arbitrary text.

Video demonstration (click the picture):

Toolbox demo

Papers implemented

URL Designation Title Implementation source
1806.04558 SV2TTS Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis This repo
1802.08435 WaveRNN (vocoder) Efficient Neural Audio Synthesis fatchord/WaveRNN
1703.10135 Tacotron (synthesizer) Tacotron: Towards End-to-End Speech Synthesis fatchord/WaveRNN
1710.10467 GE2E (encoder) Generalized End-To-End Loss for Speaker Verification This repo

Heads up

Like everything else in Deep Learning, this repo has quickly gotten old. Many SaaS apps (often paying) will give you a better audio quality than this repository will. If you wish for an open-source solution with a high voice quality:

Running the toolbox

Both Windows and Linux are supported.

  1. Install ffmpeg. This is necessary for reading audio files. Check if it's installed by running in a command line
ffmpeg
  1. Install uv for python package management
# On Windows:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
# On Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Alternatively, on any platform if you have pip installed you can do
pip install -U uv
  1. Run one of the following commands
# Run the toolbox if you have an NVIDIA GPU
uv run --extra cuda demo_toolbox.py
# Use this if you don't
uv run --extra cpu demo_toolbox.py

# Run in command line if you don't want the GUI
uv run --extra cuda demo_cli.py
uv run --extra cpu demo_cli.py

Uv will automatically create a .venv directory for you with an appropriate python environment. Open an issue if this fails for you

(Optional) Download Pretrained Models

Pretrained models are now downloaded automatically. If this doesn't work for you, you can manually download them from Hugging Face.

(Optional) Download Datasets

For playing with the toolbox alone, I only recommend downloading LibriSpeech/train-clean-100. Extract the contents as <datasets_root>/LibriSpeech/train-clean-100 where <datasets_root> is a directory of your choosing. Other datasets are supported in the toolbox, see here. You're free not to download any dataset, but then you will need your own data as audio files or you will have to record it with the toolbox.

Real-Time-Voice-Cloning (CorentinJ/Real-Time-Voice-Cloning)

Searching Database…

60.2K ·GitHub
Real-Time-Voice-Cloning
What is this?It's an implementation of SV2TTS, a deep learning framework that lets you clone a voice from just a few seconds of audio and then generate arbitrary speech in that voice in real time.
Is this a packaged app or a research toolbox?It's a research toolbox (originally the author's master's thesis) built from three separate deep learning stages: a speaker encoder, a Tacotron synthesizer, and a WaveRNN vocoder, run from the command line rather than installed as a standalone app.
How do I get the toolbox running?Install ffmpeg (needed to read audio files), install uv for Python package management, then run one of the provided commands; uv automatically creates a .venv with the right Python environment, and pretrained models download automatically.
What do I need installed before running it?You need ffmpeg for audio file reading and uv for Python package management; pretrained models are fetched automatically, with a manual Hugging Face download as a fallback.
Is this still state-of-the-art for voice cloning?No, the author says the repo has aged quickly and many paid SaaS apps now give better audio quality; he points to 'Chatterbox' as a more current open-source alternative.
Does it run on both Windows and Linux?Yes, both Windows and Linux are explicitly supported for running the toolbox.
What datasets does it work with?For experimenting with the toolbox alone the author recommends downloading LibriSpeech/train-clean-100 and extracting it locally; other datasets are also supported, or you can bring your own audio or record it directly in the toolbox.
Why does the author recommend checking other projects first?Because deep learning research moves fast and this repo is now outdated; he suggests browsing papers with code for newer speech synthesis work or trying 'Chatterbox' for 2025-level voice cloning quality.
What papers does this implement?It implements SV2TTS (speaker-verification-to-TTS transfer learning) and the GE2E speaker encoder loss itself, and reuses WaveRNN and Tacotron implementations from fatchord/WaveRNN for the vocoder and synthesizer stages.
What are similar projects?VoxCPM, TTS, MockingBird, FireRedTTS3, OpenVoice, MOSS-TTS-Nano, bark, FireRedAudio
Which topics is it in?ai-research, speech, voice-cloning, speech-synthesis, tensorflow, tts, text-to-speech, neural-network, real-time, audio, huggingface, pytorch
Creator
CorentinJ
Size
361.1 MB
Stars
60.2K
Updated
CommitMar 09
Age
7y
Commits · 90 Days