neutts README

The project's README from github.com/neuphonic/neutts. As one markdown file: README.md.

NeuTTS

HuggingFace 🤗:

NeuTTS-2E Demo Video

Created by Neuphonic - building faster, smaller, on-device voice AI

State-of-the-art Voice AI has been locked behind web APIs for too long. NeuTTS is a collection of open source, on-device, TTS speech language models with instant voice cloning. Built off of LLM backbones, NeuTTS brings natural-sounding speech, real-time performance, built-in security and speaker cloning to your local device - unlocking a new category of embedded voice agents, assistants, toys, and compliance-safe apps.

Key Features

[!CAUTION] Websites like neutts.com are popping up and they're not affliated with Neuphonic, our github or this repo.

We are on neuphonic.com only. Please be careful out there! 🙏

Model Details

NeuTTS models are built from small LLM backbones - lightweight yet capable language models optimised for text understanding and generation - as well as a powerful combination of technologies designed for efficiency and quality:

NeuTTS-Air NeuTTS-Nano Models NeuTTS-2E
# Params (Active) ~360m ~120m ~125m
# Params (Emb + Active) ~552m ~229m ~236m
Input Format Phonemes Phonemes Text
License Apache 2.0 NeuTTS Open License 1.0 NeuTTS Open License 1.0

Backbone Variants

Model PyTorch repo GGUF repos Cloning Emotions Streaming
NeuTTS-Air neuphonic/neutts-air neuphonic/neutts-air-{q4,q8}-gguf Yes — GGUF only
NeuTTS-Nano neuphonic/neutts-nano neuphonic/neutts-nano-{q4,q8}-gguf Yes — GGUF only
NeuTTS-Nano German neuphonic/neutts-nano-german neuphonic/neutts-nano-german-{q4,q8}-gguf Yes — GGUF only
NeuTTS-Nano French neuphonic/neutts-nano-french neuphonic/neutts-nano-french-{q4,q8}-gguf Yes — GGUF only
NeuTTS-Nano Spanish neuphonic/neutts-nano-spanish neuphonic/neutts-nano-spanish-{q4,q8}-gguf Yes — GGUF only
NeuTTS-2E neuphonic/neutts-2e neuphonic/neutts-2e-{q4,q8}-gguf 4 fixed speakers 6 + neutral GGUF only

Codec Variants

Codec Engine Encode Decode GPU support
neuphonic/neucodec PyTorch Yes Yes Yes
neuphonic/distill-neucodec PyTorch Yes Yes Yes
neuphonic/neucodec-onnx-decoder ONNX — Yes —
neuphonic/neucodec-onnx-decoder-int8 ONNX — Yes —

Any backbone can be paired with any codec. Decoder-only (ONNX) codecs require pre-encoded references — these ship with the repo for all bundled voices, including the NeuTTS-2E speakers.

Throughput Benchmarking

These benchmarks are for the Q4_0 quantisations neutts-air-Q4_0 and neutts-nano-Q4_0. Note that all models in the NeuTTS-Nano Multilingual Collection have an identical architecture, so these results should apply for any Q4_0 model in the collection.

CPU benchmarking used llama-bench (from llama.cpp) to measure prefill and decode throughput at multiple context sizes. For the GPU benchmark (RTX 4090), we leverage vLLM to maximise throughput, using the vLLM benchmark.

We include benchmarks on four devices: Galaxy A25 5G, AMD Ryzen 9HX 370, iMac M4 16GB, NVIDIA GeForce RTX 4090.

NeuTTS-Air NeuTTS-Nano
Galaxy A25 5G (CPU only) 20 tokens/s 45 tokens/s
AMD Ryzen 9 HX 370 (CPU only) 119 tokens/s 221 tokens/s
iMAc M4 16 GB (CPU only) 111 tokens/s 195 tokens/s
RTX 4090 16194 tokens/s 19268 tokens/s

[!NOTE] llama-bench used 14 threads for prefill and 16 threads for decode (as configured in the benchmark run) on AMD Ryzen 9HX 370 and iMac M4 16GB, and 6 threads for each on the Galaxy A25 5G. The tokens/s reported are when having 500 prefill tokens and generating 250 output tokens.

[!NOTE] Please note that these benchmarks only include the Speech Language Model and do not include the Codec which is needed for a full audio generation pipeline.

Get Started with NeuTTS

[!NOTE] We have added a streaming example using the llama-cpp-python library as well as a finetuning script. For finetuning, please refer to the finetune guide for more details.

  1. Install NeuTTS

    pip install neutts

    Or for a local editable install, clone this repository and run in the base folder:

    pip install -e .

    Alternatively to install all dependencies, including onnxruntime and llama-cpp-python (equivalent to steps 2 and 3 below):

    pip install neutts[all]

    or for an editable install:

    pip install -e .[all]

  2. (Optional) Install llama-cpp-python to use .gguf models.

    To use any of the GGUF backbones (e.g., in basic_streaming_example.py) you need to install the llama-cpp-python package.

    For the best performance, you must compile this package from source with hardware acceleration enabled for your specific operating system and target device (CPU or GPU).

    macOS (Apple Silicon)

    For M-series Macs, it is highly recommended to use Apple's native Accelerate framework for optimized CPU performance:

       CMAKE_ARGS="-DGGML_METAL=OFF -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple" pip install "neutts[llama]" --force-reinstall --no-cache-dir

    Linux (OpenBLAS)

    For Linux, you can accelerate CPU performance using OpenBLAS.

    Prerequisite: Ensure you have OpenBLAS installed on your system (e.g., sudo apt-get install libopenblas-dev on Ubuntu). For other distros, refer to the OpenBLAS Installation Guide.

       CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install "neutts[llama]" --force-reinstall --no-cache-dir

    Windows (OpenBLAS)

    Prerequisite: Ensure you have OpenBLAS installed on your system. Please refer to the OpenBLAS Installation Guide.

    For Windows users utilizing PowerShell, set the environment variable and run the install command like this:

       $env:CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"; pip install "neutts[llama]" --force-reinstall --no-cache-dir

    Looking for GPU Support?

    If you have a dedicated GPU (Nvidia/CUDA, AMD/ROCm, M-Series Mac/Metal) and want to utilize it instead of the CPU, the CMAKE flags will be different.Please refer to the official llama-cpp-python documentation for the exact flags required for your specific hardware.

  3. (Optional) Install onnxruntime to use the .onnx decoder.

    pip install "neutts[onnx]"

Examples

To get started with the example scripts, clone this repository and navigate into the project directory:

git clone https://github.com/neuphonic/neutts.git
cd neutts

Several examples are available, including a Jupyter notebook in the examples folder.

Basic Example

Run the basic example script to synthesize speech:

python -m examples.basic_example \
  --input_text "My name is Andy. I'm 25 and I just moved to London. The underground is pretty confusing, but it gets me around in no time at all." \
  --ref_audio samples/jo.wav \
  --ref_text samples/jo.txt

To specify a particular model repo for the backbone or codec, add the --backbone and --codec arguments. Available backbones are listed in the NeuTTS-Air, NeuTTS-Nano Multilingual Collection and NeuTTS-2E huggingface collections.

[!CAUTION] If you are using a non-English backbone, it is highly recommended to use a same-language reference for best performance. See the 'example reference files' section below to select an appropriate example reference.

One-Code Block Usage

from neutts import NeuTTS
import soundfile as sf

tts = NeuTTS(
   backbone_repo="neuphonic/neutts-nano", # or 'neuphonic/neutts-nano-q4-gguf' with llama-cpp-python installed
   backbone_device="cpu",
   codec_repo="neuphonic/neucodec",
   codec_device="cpu"
)
input_text = "My name is Andy. I'm 25 and I just moved to London. The underground is pretty confusing, but it gets me around in no time at all."

ref_text = "samples/jo.txt"
ref_audio_path = "samples/jo.wav"

ref_text = open(ref_text, "r").read().strip()
ref_codes = tts.encode_reference(ref_audio_path)

wav = tts.infer(input_text, ref_codes, ref_text)
sf.write("test.wav", wav, 24000)

Streaming

Speech can also be synthesised in streaming mode, where audio is generated in chunks and plays as generated. The streaming example scripts use pyaudio for playback, which needs the PortAudio library on macOS and Linux:

# macOS
brew install portaudio && pip install pyaudio

# Ubuntu/Debian
sudo apt install portaudio19-dev && pip install pyaudio

# Windows (prebuilt wheels)
pip install pyaudio

Then run:

python -m examples.basic_streaming_example \
  --input_text "My name is Andy. I'm 25 and I just moved to London. The underground is pretty confusing, but it gets me around in no time at all." \
  --ref_codes samples/jo.pt \
  --ref_text samples/jo.txt

Again, a particular model repo can be specified with the --backbone argument - note that for streaming the model must be in GGUF format.

Emotional TTS (NeuTTS-2E)

NeuTTS-2E is a fixed-speaker emotional English model. Pre-encoded references for its four speakers (emily, paul, sophie, steven) live in samples/ in the same format as every other reference, so no reference audio is needed — and they work as ordinary cloning references for the other models too. Supported emotions are angry, disgusted, fearful, happy, neutral, sad and surprised.

from neutts import NeuTTS2E
import soundfile as sf

tts = NeuTTS2E()
wav = tts.infer("I can't believe it's finally here!", speaker="emily", emotion="happy")
sf.write("test.wav", wav, 24000)

Or from the command line:

python -m examples.basic_example_emotions \
  --input_text "I can't believe it's finally here!" \
  --speaker emily \
  --emotion happy

Streaming works the same way via examples.basic_streaming_example_emotions, which uses the GGUF backbones (neuphonic/neutts-2e-q8-gguf by default, or neuphonic/neutts-2e-q4-gguf via --backbone).

Reproducibility

Generation is sampled, and every call prints the seed it used. With no seed set, each call draws a fresh random seed, so repeated calls give different takes — to recover a take you liked, rerun with its printed seed. Passing seed to NeuTTS/NeuTTS2E (or --seed to the examples) pins it: the same inputs and seed always produce identical audio, on both the PyTorch and GGUF backbones, batch or streaming.

Preparing References for Cloning

NeuTTS requires two inputs:

  1. A reference audio sample (.wav file)
  2. A text string

The model then synthesises the text as speech in the style of the reference audio. This is what enables NeuTTS models' instant voice cloning capability.

Example Reference Files

You can find some ready-to-use references in the samples folder:

Guidelines for Best Results

For optimal performance, reference audio samples should be:

  1. Mono channel
  2. 16-44 kHz sample rate
  3. 3–15 seconds in length
  4. Saved as a .wav file
  5. Clean — minimal to no background noise
  6. Natural, continuous speech — like a monologue or conversation, with few pauses, so the model can capture tone effectively

Guidelines for minimizing Latency

For optimal performance on-device:

  1. Use the GGUF model backbones
  2. Pre-encode references (see examples/encode_reference.py or examples/basic_example.py)
  3. Use the onnx codec decoder

Take a look at this example in the examples README to get started.

Responsibility

Every audio file generated by NeuTTS includes by default a Perth (Perceptual Threshold) Watermark.

Note: If you install neutts using uv sync within the repo, the program will still run, but watermarking will be disabled (you will see warning that perth is missing). This is because uv sync currently fails to pull the required Perth dependencies, please see This Issue. To ensure watermarking is active, please install the package via PyPI instead (pip install neutts).

Disclaimer

Don't use this model to do bad things… please.

Developer Requirements

To run the pre commit hooks to contribute to this project run:

pip install pre-commit

Then:

pre-commit install

Running Tests

First, install the dev requirements:

pip install -r requirements-dev.txt

To run the tests:

pytest tests/

To test loading of all the official backbone and codecs, use:

RUN_SLOW=true pytest tests/

neutts (neuphonic/neutts)

Searching Database…

6.3K ·GitHub
Neutts
这是什么？NeuTTS 是一组开源的、可在本地设备上运行的文本转语音语言模型，只需几秒钟的参考音频就能克隆出一个声音。
这是一个拿来直接运行的现成模型，还是需要用来搭建应用的框架？它是一个 Python 软件包，外加一系列预训练模型检查点（PyTorch 和 GGUF 格式）。你需要先安装它，然后在脚本中调用，或使用命令行示例。
怎样才能让它跑起来？安装软件包（使用 pip install neutts 安装带水印的 PyPI 版本），克隆代码仓库以获取示例脚本，然后用一个参考 .wav 文件和一段文本运行基础示例，即可合成语音。
除了 pip install 之外，使用 GGUF 模型还需要别的东西吗？需要。要运行 .gguf 格式的骨干模型，你必须安装 llama-cpp-python，最好从源码编译并启用硬件加速（macOS 上用 Accelerate，Linux/Windows 上用 OpenBLAS）；如果想使用 ONNX 音频解码器，还需要安装 onnxruntime。
水印或语音克隆有什么需要注意的坑吗？如果你通过 uv sync 安装，音频水印会被悄悄禁用，因为 uv sync 目前无法拉取 Perth 依赖，所以要确保水印生效，就需要使用 PyPI 安装；另外，参考音频与模型语言一致时，克隆效果最好。
它真的能在手机或树莓派上运行，而不只是电脑上吗？可以。基准测试涵盖了 Galaxy A25 5G 手机、树莓派、AMD Ryzen 笔记本、Apple M4 Mac 和 RTX 4090 显卡，均使用量化后的 GGUF 模型。
生成语音需要联网或调用云端 API 吗？不需要。模型下载完成后即可完全在本地设备上运行，这正是该项目相较于基于网页的 TTS API 的核心优势。
这些模型能说哪些语言？所有模型系列都支持英语；NeuTTS-Nano 多语言系列及其各语言专用的 GGUF 版本还支持西班牙语、德语和法语。
在真实硬件上生成速度有多快？在仅使用 CPU 的中端设备上，NeuTTS-Air 大约可达到 20–119 tokens/s，NeuTTS-Nano 为 45–221 tokens/s，具体取决于设备；而通过 vLLM 使用 RTX 4090 显卡时，速度超过 16,000 tokens/s。
语音克隆到底是怎么实现的？你提供一段简短（3–15 秒）的单声道 .wav 参考音频和一段文本，模型就会用参考声音的风格合成这段文本。其实现方式是 LLM 骨干网络配合 NeuCodec 神经音频编解码器。
NeuTTS-2E 是什么？NeuTTS-2E 是一个固定说话人的英语模型，增加了情感控制功能，内置四位说话人，支持七种情绪，包括愤怒、高兴和悲伤。
What are similar projects?VoxCPM, MOSS-TTS, pocket-tts, MOSS-TTS-Nano, sherpa-onnx, OmniVoice, Index-Translate, bark
Which topics is it in?on-device, voice-assistant, voice-cloning, speech-synthesis, gguf, onnx, raspberry-pi, text-to-speech, embedded, quantization, multilingual, streaming
Creator
neuphonic
Size
8.8 MB
Stars
6.3K
Updated
CommitJul 22
Age
1y
Commits · 90 Days