nari-qwen3-tts README

The project's README from github.com/nari-labs/nari-qwen3-tts. As one markdown file: README.md.

Nari Qwen3-TTS

TL;DR

Nari Qwen3-TTS is a high-performance, single-H100 serving implementation of Qwen3-TTS 1.7B CustomVoice. It exposes streaming and non-streaming speech generation over HTTP, plus WebSocket-based input streaming for incremental text input.

It achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on a single NVIDIA H100 SXM. Even at 20 RPS, it sustains sub-80 ms p95 TTFA.

Below is a performance comparison with popular serving engines.

p95 TTFA under load comparison

[!NOTE] Methodology

Read our blog post for the full methodology. For details on the benchmark, see the benchmark repository.

Run with Docker

The container requires an NVIDIA H100, the NVIDIA Container Toolkit, and a driver compatible with CUDA 13.0. The published image supports Linux x86_64 (linux/amd64) H100 hosts. The engine has been tested only with English as the primary language.

docker run --rm --gpus all \
  -p 8000:8000 \
  -e HF_TOKEN \
  -e QWEN3_TTS_PROFILE=ttfa \
  -v nari-qwen3-tts-cache:/home/nari/.cache \
  ghcr.io/nari-labs/nari-qwen3-tts:latest

Model files and compiled kernels are cached in the named volume. Once model loading and CUDA Graph capture finish, check readiness with:

curl --fail http://127.0.0.1:8000/ready

To build the image locally instead:

docker build \
  --build-arg VCS_REF="$(git rev-parse HEAD)" \
  -t nari-qwen3-tts:local .

Run with uv

Python and uv versions are pinned in .python-version and pyproject.toml. The checked-in uv.lock defines the complete environment.

Install the required system packages (Debian/Ubuntu):

sudo apt-get install -y build-essential libsndfile1 sox

uv sync --frozen --extra codec --extra cuda --extra serving
uv run --frozen nari-qwen3-tts-server --profile ttfa

Add --local-files-only after the model is cached to prevent downloads at startup. --frozen is intentional: an out-of-date lockfile fails instead of silently resolving a different CUDA or Python environment.

The distribution name uses hyphens, while Python imports use underscores:

from nari_qwen3_tts import ModelAssetConfig, open_model

Profiles

Select a Docker profile with QWEN3_TTS_PROFILE, or pass --profile to nari-qwen3-tts-server. You can also set QWEN3_TTS_MODEL_CACHE_DIR and QWEN3_TTS_LOCAL_FILES_ONLY=1 in Docker.

For advanced tuning, apply a strict partial YAML overlay:

nari-qwen3-tts-server \
  --local-files-only \
  --engine-config /path/to/engine.yaml

The overlay may name its packaged base with extends: ttfa, balanced, or throughput; alternatively, pass --profile and omit extends. Unknown keys, invalid capture lists, and profile/base mismatches fail before model loading. The fully resolved config and its SHA-256 are printed at startup.

API and architecture

The service exposes:

See WebSocket speech API for the live-text protocol and client example.

HTTP client example

POST /v1/audio/speech follows the OpenAI Audio Speech request shape. Nari Qwen3-TTS supports only the model listed above, wav and pcm output, and speed: 1.0. It also accepts Nari-specific controls such as language.

curl http://127.0.0.1:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    "input": "Hello from Nari Labs.",
    "voice": "ryan",
    "language": "english",
    "response_format": "wav",
    "stream": false
  }' \
  --output speech.wav

Readiness remains false until CUDA Graph capture and a warm-up TTS request have both completed.

Thanks and references

Built by Nari Labs. Thanks to the Qwen3-TTS authors for releasing Qwen3-TTS, and to these projects for their work on high-performance multimodal and speech serving:

nari-qwen3-tts (nari-labs/nari-qwen3-tts)

Searching Database…

187 ·GitHub
Nari-qwen3-tts
What is this?Nari Qwen3-TTS is a high-performance serving implementation of the Qwen3-TTS 1.7B CustomVoice model, built to run on a single NVIDIA H100 and exposed as an API with streaming and non-streaming speech generation.
Is this a model file I download, or a server I run?It is a server/API, not a standalone model file: it exposes text-to-speech generation over HTTP and WebSocket endpoints such as POST /v1/audio/speech and WS /v1/audio/speech/ws.
How do I get it running?Pull and run the published Docker image on an H100 host, or install locally with uv using the pinned .python-version, pyproject.toml, and uv.lock files; model files and compiled CUDA kernels are cached in a named volume.
Do I need a specific GPU to run this?Yes, it requires a single NVIDIA H100, the NVIDIA Container Toolkit, and a driver compatible with CUDA 13.0; the published Docker image only targets Linux x86-64 H100 hosts.
Does it support languages other than English?It has only been tested with English as the primary language, so other languages are not guaranteed to work well.
What scheduling profiles can I pick?Three built-in profiles are offered: 'ttfa' optimizes for lowest first-audio latency with smaller chunks, 'balanced' is the default, and 'throughput' uses larger batches for higher aggregate load; select via QWEN3_TTS_PROFILE or --profile.
Is the API compatible with OpenAI's TTS endpoint?POST /v1/audio/speech follows the OpenAI Audio Speech request shape, though this server only supports its single bundled model, wav/pcm output, and speed 1.0, plus Nari-specific controls like language.
How fast is the speech generation?On a single H100 SXM it sustains 10 requests per second with sub-50ms p95 time-to-first-audio, and even at 20 RPS it holds sub-80ms p95 TTFA while keeping real-time playback.
How does it compare to other TTS serving engines?The README benchmarks it against other high-performance serving engines like vLLM-Omni, SGLang-Omni and VoxServe, pointing to a separate benchmark repository and blog post for the full methodology.
What are similar projects?vllm-omni, HyperQwen, InstinctFlash, TensorFold, text-generation-inference, csm, project-maya, dgx-spark-qwen38
Which topics is it in?streaming-audio, inference-optimization, http-server, inference-engine, model-serving, audio-generation, text-to-speech, websocket, openai-compatible, streaming, inference, nvidia
Creator
nari-labs
Size
0.5 MB
Stars
187
Updated
CommitAug 19
Age
2mo
Commits · 90 Days