fish-speech README

The project's README from github.com/fishaudio/fish-speech. As one markdown file: README.md.

Fish Speech

English | 简体中文 | Portuguese | 日本語 | 한국어 | العربية | Español

Fish Audio S1 - Expressive Voice Cloning and Text-to-Speech | Product Hunt fishaudio%2Ffish-speech | Trendshift




Discord Docker QQ Channel
HuggingFace Model Fish Audio Blog Paper | Technical Report

[!IMPORTANT] License Notice
This codebase and its associated model weights are released under FISH AUDIO RESEARCH LICENSE. Please refer to LICENSE for more details. We will take action against any violation of the license.

[!WARNING] Legal Disclaimer
We do not hold any responsibility for any illegal usage of the codebase. Please refer to your local laws about DMCA and other related laws.

Quick Start

For Human

Here are the official documents for Fish Audio S2, follow the instructions to get started easily.

[!IMPORTANT] For SGLang server, please read SGLang-Omni README.

For vLLM Omni server, please read vLLM-Omni Fish Speech S2 Pro Recipe and the User Guide.

For LLM Agent

Install and configure Fish-Audio S2 by following the instructions here: https://speech.fish.audio/install/

Fish Audio S2 Pro

State-of-the-art multilingual text-to-speech (TTS) system, redefining the boundaries of voice generation.

Fish Audio S2 Pro is the most advanced multimodal model developed by Fish Audio. Trained on over 10 million hours of audio data covering more than 80 languages, S2 Pro combines a Dual-Autoregressive (Dual-AR) architecture with reinforcement learning (RL) alignment to generate speech that is exceptionally natural, realistic, and emotionally rich, leading the competition among both open-source and closed-source systems.

The core strength of S2 Pro lies in its support for sub-word level fine-grained control of prosody and emotion using natural language tags (e.g., [whisper], [excited], [angry]), while natively supporting multi-speaker and multi-turn conversation generation.

Visit the Fish Audio website for a live playground, or read our technical report and blog post for more details.

Model Variants

Model Size Availability Description
S2-Pro 4B parameters HuggingFace Full-featured flagship model with maximum quality and stability

More details of the model can be found in the technical report.

Benchmark Results

Benchmark Fish Audio S2
Seed-TTS Eval — WER (Chinese) 0.54% (best overall)
Seed-TTS Eval — WER (English) 0.99% (best overall)
Audio Turing Test (with instruction) 0.515 posterior mean
EmergentTTS-Eval — Win Rate 81.88% (highest overall)
Fish Instruction Benchmark — TAR 93.3%
Fish Instruction Benchmark — Quality 4.51 / 5.0
Multilingual (MiniMax Testset) — Best WER 11 of 24 languages
Multilingual (MiniMax Testset) — Best SIM 17 of 24 languages

On Seed-TTS Eval, S2 achieves the lowest WER among all evaluated models including closed-source systems: Qwen3-TTS (0.77/1.24), MiniMax Speech-02 (0.99/1.90), Seed-TTS (1.12/2.25). On the Audio Turing Test, 0.515 surpasses Seed-TTS (0.417) by 24% and MiniMax-Speech (0.387) by 33%. On EmergentTTS-Eval, S2 achieves particularly strong results in paralinguistics (91.61% win rate), questions (84.41%), and syntactic complexity (83.39%).

Highlights

Fine-Grained Inline Control via Natural Language

S2 Pro brings unprecedented "soul" to speech. Using simple [tag] syntax, you can precisely embed emotional instructions at any position in the text.

Innovative Dual-Autoregressive (Dual-AR) Architecture

S2 Pro adopts a master-slave Dual-AR architecture consisting of a decoder-only transformer and an RVQ audio codec (10 codebooks, ~21 Hz):

This asymmetric design achieves peak audio fidelity while significantly boosting inference speed.

Reinforcement Learning (RL) Alignment

S2 Pro utilizes Group Relative Policy Optimization (GRPO) for post-training alignment. We use the same model suite for data cleaning and annotation directly as Reward Models, perfectly resolving the distribution mismatch between pre-training data and post-training objectives.

Extreme Streaming Performance (Powered by SGLang)

As the Dual-AR architecture is structurally isomorphic to standard LLMs, S2 Pro natively supports all SGLang inference acceleration features, including Continuous Batching, Paged KV Cache, CUDA Graph, and RadixAttention-based Prefix Caching.

Performance on a single NVIDIA H200 GPU:

Robust Multilingual Support

S2 Pro supports over 80 languages without requiring phonemes or language-specific preprocessing:

Native Multi-Speaker Generation

Fish Audio S2 allows users to upload reference audio containing multiple speakers, and the model processes each speaker's features via the <|speaker:i|> token. You can then control the model's performance via speaker ID tokens, enabling a single generation to include multiple speakers. There is no longer a need to upload separate reference audio for each individual speaker.

Multi-Turn Generation

Thanks to the expansion of the model context, our model can now leverage previous information to improve the expressiveness of subsequent generated content, thereby increasing the naturalness of the dialogue.

Rapid Voice Cloning

Fish Audio S2 supports accurate voice cloning using short reference samples (typically 10-30 seconds). The model captures timbre, speaking style, and emotional tendencies, producing realistic and consistent cloned voices without additional fine-tuning. For SGLang Server usage, please refer to the SGLang-Omni README.

Credits

Tech Report

@misc{fish-speech-v1.4,
      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
      year={2024},
      eprint={2411.01156},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2411.01156},
}

@misc{liao2026fishaudios2technical,
      title={Fish Audio S2 Technical Report}, 
      author={Shijia Liao and Yuxuan Wang and Songting Liu and Yifan Cheng and Ruoyi Zhang and Tianyu Li and Shidong Li and Yisheng Zheng and Xingwei Liu and Qingzheng Wang and Zhizhuo Zhou and Jiahua Liu and Xin Chen and Dawei Han},
      year={2026},
      eprint={2603.08823},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2603.08823}, 
}

fish-speech (fishaudio/fish-speech)

Searching Database…

33.2K ·GitHub
Fish-speech
Что это такое?Fish Speech (Fish Audio S2) — современная система синтеза речи с открытым исходным кодом, которая генерирует естественную, эмоционально выразительную речь на более чем 80 языках. Флагманская модель S2-Pro сочетает архитектуру Dual-AR с выравниванием на основе обучения с подкреплением.
Это отдельный файл модели или полноценная система со своим интерфейсом?Это полноценная система синтеза речи: её можно запустить через инструмент командной строки, WebUI, выделенный сервер инференса или внутри Docker-контейнера.
Как это запустить?Сначала следуйте инструкции по установке, затем выберите Command Line Inference, WebUI Inference или Server Inference; для запуска в контейнере также есть руководство по Docker Setup.
Нужна ли мощная видеокарта, чтобы это запустить?Да — это модель с 4 млрд параметров, рассчитанная на видеокарты NVIDIA с поддержкой CUDA. Опубликованные показатели измерялись на одной NVIDIA H200, а для высокопроизводительной работы сервера используется ускорение через SGLang.
Есть ли подвох, прежде чем внедрять это в продукт?Проект прямо снимает с себя ответственность за незаконное использование и предупреждает, что будет применять свою лицензию против нарушений, поэтому перед коммерческим использованием внимательно проверьте условия лицензии.
Что особенного в системе эмоциональных тегов?Вы вставляете в текст теги на естественном языке, например [whisper], [angry] или [professional broadcast tone], и они напрямую управляют интонацией и эмоциями. Помимо фиксированных пресетов поддерживаются более 15 000 свободных тегов.
Сколько эталонного аудио нужно для клонирования голоса?Достаточно 10–30 секунд эталонного аудио, чтобы модель смогла перенять тембр, манеру речи и эмоциональные особенности без дополнительного дообучения.
Может ли она сгенерировать диалог нескольких говорящих за один проход?Да — вы можете загрузить эталонное аудио с несколькими говорящими и управлять выводом для каждого из них с помощью токенов идентификаторов говорящих, так что одна генерация может включать несколько голосов.
Почему она так быстра для модели с 4 млрд параметров?Она использует асимметричную архитектуру Dual-AR (медленный AR на 4 млрд параметров и быстрый AR на 400 млн) вместе с ускорением SGLang (непрерывный батчинг, paged KV cache, RadixAttention). Это даёт около 100 мс до первого звука и более 3 000 акустических токенов в секунду на H200.
What are similar projects?Speech, csm, FireRedAudio, breeze-tts, MOSS-TTS, bark, VoxCPM, Retrieval-based-Voice-Conversion-WebUI
Which topics is it in?vits, voice-cloning, speech-synthesis, llama, webui, tts, text-to-speech, reinforcement-learning, multimodal, transformer, multilingual, real-time
Creator
fishaudio
Size
32.4 MB
Stars
33.2K
Updated
CommitOct 14Releasev1.5.1May 31
Age
3y
Commits · 90 Days