nari-qwen3-tts README

The project's README from github.com/nari-labs/nari-qwen3-tts. As one markdown file: README.md.

Nari Qwen3-TTS

TL;DR

Nari Qwen3-TTS is a high-performance, single-H100 serving implementation of Qwen3-TTS 1.7B CustomVoice. It exposes streaming and non-streaming speech generation over HTTP, plus WebSocket-based input streaming for incremental text input.

It achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on a single NVIDIA H100 SXM. Even at 20 RPS, it sustains sub-80 ms p95 TTFA.

Below is a performance comparison with popular serving engines.

p95 TTFA under load comparison

[!NOTE] Methodology

Read our blog post for the full methodology. For details on the benchmark, see the benchmark repository.

Run with Docker

The container requires an NVIDIA H100, the NVIDIA Container Toolkit, and a driver compatible with CUDA 13.0. The published image supports Linux x86_64 (linux/amd64) H100 hosts. The engine has been tested only with English as the primary language.

docker run --rm --gpus all \
  -p 8000:8000 \
  -e HF_TOKEN \
  -e QWEN3_TTS_PROFILE=ttfa \
  -v nari-qwen3-tts-cache:/home/nari/.cache \
  ghcr.io/nari-labs/nari-qwen3-tts:latest

Model files and compiled kernels are cached in the named volume. Once model loading and CUDA Graph capture finish, check readiness with:

curl --fail http://127.0.0.1:8000/ready

To build the image locally instead:

docker build \
  --build-arg VCS_REF="$(git rev-parse HEAD)" \
  -t nari-qwen3-tts:local .

Run with uv

Python and uv versions are pinned in .python-version and pyproject.toml. The checked-in uv.lock defines the complete environment.

Install the required system packages (Debian/Ubuntu):

sudo apt-get install -y build-essential libsndfile1 sox

uv sync --frozen --extra codec --extra cuda --extra serving
uv run --frozen nari-qwen3-tts-server --profile ttfa

Add --local-files-only after the model is cached to prevent downloads at startup. --frozen is intentional: an out-of-date lockfile fails instead of silently resolving a different CUDA or Python environment.

The distribution name uses hyphens, while Python imports use underscores:

from nari_qwen3_tts import ModelAssetConfig, open_model

Profiles

Select a Docker profile with QWEN3_TTS_PROFILE, or pass --profile to nari-qwen3-tts-server. You can also set QWEN3_TTS_MODEL_CACHE_DIR and QWEN3_TTS_LOCAL_FILES_ONLY=1 in Docker.

For advanced tuning, apply a strict partial YAML overlay:

nari-qwen3-tts-server \
  --local-files-only \
  --engine-config /path/to/engine.yaml

The overlay may name its packaged base with extends: ttfa, balanced, or throughput; alternatively, pass --profile and omit extends. Unknown keys, invalid capture lists, and profile/base mismatches fail before model loading. The fully resolved config and its SHA-256 are printed at startup.

API and architecture

The service exposes:

See WebSocket speech API for the live-text protocol and client example.

HTTP client example

POST /v1/audio/speech follows the OpenAI Audio Speech request shape. Nari Qwen3-TTS supports only the model listed above, wav and pcm output, and speed: 1.0. It also accepts Nari-specific controls such as language.

curl http://127.0.0.1:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    "input": "Hello from Nari Labs.",
    "voice": "ryan",
    "language": "english",
    "response_format": "wav",
    "stream": false
  }' \
  --output speech.wav

Readiness remains false until CUDA Graph capture and a warm-up TTS request have both completed.

Thanks and references

Built by Nari Labs. Thanks to the Qwen3-TTS authors for releasing Qwen3-TTS, and to these projects for their work on high-performance multimodal and speech serving:

nari-qwen3-tts (nari-labs/nari-qwen3-tts)

Searching Database…

187 ·GitHub
Nari-qwen3-tts
这是什么？Nari Qwen3-TTS 是 Qwen3-TTS 1.7B CustomVoice 模型的高性能服务化实现，专为在单张 NVIDIA H100 上运行而构建，以 API 形式提供流式和非流式语音生成。
这是我要下载的模型文件，还是要运行的服务器？它是一个服务器/API，不是独立的模型文件：它通过 HTTP 和 WebSocket 端点（例如 POST /v1/audio/speech 和 WS /v1/audio/speech/ws）提供文本转语音生成。
怎么把它跑起来？在 H100 主机上拉取并运行已发布的 Docker 镜像；或者用 uv 在本地安装，使用锁定的 .python-version、pyproject.toml 和 uv.lock 文件。模型文件和编译好的 CUDA 内核缓存在一个命名卷里。
运行它需要特定的 GPU 吗？需要。它要求一张 NVIDIA H100、NVIDIA Container Toolkit，以及兼容 CUDA 13.0 的驱动；已发布的 Docker 镜像只面向 Linux x86-64 的 H100 主机。
它支持英语以外的语言吗？它只以英语为主要语言做过测试，其他语言不保证效果良好。
可以选择哪些调度配置（profile）？提供三个内置配置：“ttfa”用更小的分块优化最低的首音频延迟，“balanced”是默认值，“throughput”用更大的批次承载更高的聚合负载；通过 QWEN3_TTS_PROFILE 或 --profile 选择。
这个 API 与 OpenAI 的 TTS 端点兼容吗？POST /v1/audio/speech 遵循 OpenAI Audio Speech 的请求结构，但该服务器只支持其自带的单一模型、wav/pcm 输出和 1.0 倍速，外加 language 等 Nari 特有的控制项。
语音生成有多快？在单张 H100 SXM 上，它能持续承受每秒 10 个请求，p95 首音频延迟低于 50 毫秒；即使到每秒 20 个请求，p95 TTFA 仍保持在 80 毫秒以内，同时维持实时播放。
与其他 TTS 服务引擎相比它怎么样？README 把它与 vLLM-Omni、SGLang-Omni 和 VoxServe 等其他高性能服务引擎做了基准对比，完整的方法论见单独的基准仓库和博客文章。
What are similar projects?vllm-omni, HyperQwen, InstinctFlash, TensorFold, text-generation-inference, csm, project-maya, dgx-spark-qwen38
Which topics is it in?streaming-audio, inference-optimization, http-server, inference-engine, model-serving, audio-generation, text-to-speech, websocket, openai-compatible, streaming, inference, nvidia
Creator
nari-labs
Size
0.5 MB
Stars
187
Updated
CommitAug 19
Age
2mo
Commits · 90 Days