AuK README

The project's README from github.com/Tencent-Hunyuan/AuK. As one markdown file: README.md.

AuK: An Open-Source Foundational Model for Speech Generation and Editing

💻 Try our model on the HuggingFace Space · ModelScope Space!

News

Demo

English

https://github.com/user-attachments/assets/d07332fc-5f69-4f16-9d00-a7443cc19e6a

中文

https://github.com/user-attachments/assets/c532bbdb-e6ce-4434-a9a5-16f29a8d4135

Contents

Introduction

AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface. AuK has two variants:

Model Description Weight
AuK Base model for high-quality generation 🤗 Hugging Face · 🤖 ModelScope
AuK-Flash Distilled model for fast 4-step inference 🤗 Hugging Face · 🤖 ModelScope

Performance

AuK performance across speech generation, editing, enhancement, and separation benchmarks

Model Architecture

Model Architecture

Supported Tasks

AuK exposes every task through the same natural-language instruction interface. The table below groups the supported tasks by category, with a short description and a link to its section in the Cookbook, where you'll find instruction templates plus CLI and Python examples.

Category Task Description Cookbook
Speech Generation Zero-shot TTS Speak the target text in the voice of the reference audio. Zero-shot TTS
Instruct TTS Generate speech from a voice description alone — no reference audio. Instruct TTS
Content Editing Speech Content Editing Rewrite what is said — replace, insert, or remove text. Speech Content Editing
Lyric Editing Rewrite lyrics in a singing recording while preserving the melody and voice. Lyric Editing
Acoustic Editing Pitch Editing Raise or lower the pitch by semitones. Pitch Editing
Speed Editing Adjust the speaking rate; output length scales with the speed factor. Speed Editing
Volume Editing Raise or lower the volume by decibels. Volume Editing
Paralinguistic Editing Emotion Change the emotion while preserving content and voice. Emotion
Timbre Change the timbre to a description while keeping the content unchanged. Timbre
De-accent Remove a regional accent while preserving the speaker's voice and content. De-accent
Nonverbal Editing Remove or add nonverbal sounds such as breaths, laughs, or coughs. Nonverbal Editing
Whisper Conversion Convert between normal speech and whisper while preserving speaker and content. Whisper Conversion
Enhancement & Separation Speech Enhancement Denoise, dereverberate, or restore natural, clear speech. Speech Enhancement
Speech Separation Keep one speaker by talking order and remove the others. Speech Separation
Music Separation Extract the singing voice from a mix, or keep all human voices. Music Separation
Target Speaker Extraction Keep the target speaker identified by what they say. Target Speaker Extraction

Quick Start

Installation

Clone the repository, then choose either uv or Conda to create an isolated Python 3.10 environment.

git clone https://github.com/Tencent-Hunyuan/AuK
cd AuK

uv

# Create and activate a project-local environment.
uv venv --python 3.10
source .venv/bin/activate

# Choose one installation target:
# Core inference and CLI only
uv pip install -e .

# Core inference + Gradio + Prompt Enhancer + ASR
uv pip install -e ".[gradio]"

# Core inference + ComfyUI nodes + Prompt Enhancer + ASR
uv pip install -e ".[comfyui]"

# Core inference + fine-tuning
uv pip install -e ".[train]"

# Everything
uv pip install -e ".[gradio,train]"

Conda

conda create -n auk python=3.10 -y
conda activate auk

# Choose one installation target:
# Core inference and CLI only
pip install -e .

# Core inference + Gradio + Prompt Enhancer + ASR
pip install -e ".[gradio]"

# Core inference + ComfyUI nodes + Prompt Enhancer + ASR
pip install -e ".[comfyui]"

# Core inference + fine-tuning
pip install -e ".[train]"

# Everything
pip install -e ".[gradio,train]"

The default installation includes PyTorch, TorchAudio, and TorchVision. If your platform requires a specific CPU or CUDA build, install a matching PyTorch stack for your platform first, then install AuK with either command above.

Download the weights

🤗 HuggingFace

pip install -U "huggingface_hub[cli]"

# AuK-Base
hf download tencent/AuK --local-dir ./ckpts/AuK

# AuK-Flash (4-step distilled) 
hf download tencent/AuK-Flash --local-dir ./ckpts/AuK-Flash

# MLLM Encoder
hf download Qwen/Qwen2.5-Omni-3B  --local-dir ./ckpts/Qwen2.5-Omni-3B

🤖 ModelScope

pip install -U modelscope

# AuK-Base
modelscope download --model Tencent-Hunyuan/AuK --local_dir ./ckpts/AuK

# AuK-Flash (4-step distilled)
modelscope download --model Tencent-Hunyuan/AuK-Flash  --local_dir ./ckpts/AuK-Flash

# MLLM Encoder
modelscope download --model Qwen/Qwen2.5-Omni-3B --local_dir ./ckpts/Qwen2.5-Omni-3B

The expected directory structure is:

ckpts/
├── AuK/
├── AuK-Flash/          # optional
└── Qwen2.5-Omni-3B/

The model checkpoint contains the diffusion transformer and layer-fusion weights. The MLLM encoder and VAE are loaded from separate files at runtime, so missing text_encoder.* keys during checkpoint loading are expected.

Command-line inference

All tasks use the same message-based interface. An --instruction is always required, while source or reference --audio is optional depending on the task. The examples below are just a taste — for the full instruction templates and per-task CLI examples, see the Cookbook.

[!TIP] When starting from a free-form request, we recommend using Prompt Enhancer. It prepares the model instruction, target duration, and any required audio preprocessing, then prints a ready-to-run one-line auk-infer command.

Content editing

Rewrite what is said by describing the change in the instruction:

auk-infer \
    --audio assets/demo-input-audio/content-edit/content.wav \
    --instruction "Replace 'but accepting what we cannot have' with 'and living well with dreams unmet'." \
    --output out_content_edit.wav \
    --gen_seconds 7.0

Speech enhancement / separation

Denoising, enhancement, and source separation are the same message-driven call — just say what to keep or remove:

auk-infer \
    --audio assets/demo-input-audio/vocal-extraction/vocal-1-input.wav \
    --instruction "请将这段音频恢复成纯净人声版本：保留原本所有说话人，并去除其中的噪声和混响，输出等长的纯净语音。" \
    --output out_denoise.wav

Zero-shot TTS

Write the target text into the instruction, then hint the duration with --gen_text (+ optional --ref_text, the reference transcript) or an explicit --gen_seconds:

auk-infer \
    --audio assets/demo-input-audio/zero-shot-tts/ref.wav \
    --instruction "Say the following with the same voice: 'Ladies and gentlemen, it's an honor to have the opportunity to address such a distinguished audience'" \
    --output out_tts.wav \
    --gen_seconds 6.0

To use AuK-Flash, set:

--ckpt ckpts/AuK-Flash/auk_flash.safetensors

AuK-Flash use 4 fixed time steps and set CFG=0.

Interactive Gradio demo

Install the Gradio dependencies with pip install -e ".[gradio]" (or the equivalent uv pip install command above) before starting the demo.

Prompt Enhancer requires an OpenAI-compatible LLM:

Export the credentials before starting Gradio:

# Required when Prompt Enhancer is enabled
export LLM_API_KEY="your-llm-api-key"
export LLM_BASE_URL="https://tokenhub.tencentmaas.com/v1"
export LLM_MODEL_NAME="hy3"

# Optional cloud ASR; omit these to use local SenseVoiceSmall
export TENCENTCLOUD_SECRET_ID="your-tencentcloud-secret-id"
export TENCENTCLOUD_SECRET_KEY="your-tencentcloud-secret-key"
export ASR_ENGINE_MODEL_TYPE="16k_zh_en"

Alternatively, copy the example file and load it:

cp .env.example .env
set -a
source ./.env
set +a

LLM_BASE_URL should be the API root, without /chat/completions. For audio-backed PE tasks, Tencent Cloud recording-file recognition is used when its credentials are configured; if it is unavailable, AuK automatically downloads and lazily loads iic/SenseVoiceSmall as a CPU fallback. The gradio extra includes both cloud and local ASR dependencies.

Choose the model according to your needs:

Recommended full command for loading both models on two GPUs:

auk-gradio \
  --base_ckpt ckpts/AuK/auk_base.safetensors \
  --flash_ckpt ckpts/AuK-Flash/auk_flash.safetensors \
  --base_config ckpts/AuK/config.yaml \
  --flash_config ckpts/AuK-Flash/config.yaml \
  --qwen_path ckpts/Qwen2.5-Omni-3B \
  --base_device cuda:0 \
  --flash_device cuda:1 \
  --dtype bf16 \
  --host 0.0.0.0 \
  --port 8080 \
  --preload

The main options above mean:

With the standard directory layout, the shortest command detects every installed variant under ckpts/AuK and ckpts/AuK-Flash:

auk-gradio

To expose only one model, pass only its checkpoint:

# Base only
auk-gradio \
  --base_ckpt ckpts/AuK/auk_base.safetensors \
  --base_device cuda:0

# Flash only
auk-gradio \
  --flash_ckpt ckpts/AuK-Flash/auk_flash.safetensors \
  --flash_device cuda:0

To load both models on one GPU:

auk-gradio \
  --base_device cuda:0 \
  --flash_device cuda:0 \
  --preload

Once either checkpoint option is supplied, only explicitly supplied variants are shown. The VAE is automatically loaded from vae.safetensors next to each checkpoint. The current implementation uses Qwen2.5-Omni-3B; Qwen3-Omni is not currently supported by --qwen_path.

Other examples:

auk-gradio --share
auk-gradio --port 8000

ComfyUI

Use AuK Base and AuK-Flash for speech generation, editing, enhancement, and separation through AuK Model Loader and AuK Generate / Edit. Install .[comfyui] in the environment that runs ComfyUI, link comfyui/ComfyUI-AuK into ComfyUI/custom_nodes, and open the reusable auk.json workflow.

The included workflow starts with Base, PE disabled, and a 3-second text-only example. See the ComfyUI guide for installation, shared .env configuration, Flash settings, audio input/output, and the integration's 30-second source-plus-target sequence limit.

Prompt Enhancer

PE uses the same OpenAI-compatible LLM environment variables described above. Load them from .env, then run:

set -a
source ./.env
set +a

python src/auk/infer/pe.py \
  --audio assets/demo-input-audio/whisper/wh-w2n-zh-input.wav \
  --instruction "Convert this whisper into normal speech while preserving the speaker and content." \
  --asr auto

PE prints the generated command:

auk-infer \
  --audio assets/after_pe/wh-w2n-zh-input.wav \
  --instruction 'Convert this whispered speech into normal speech.' \
  --output assets/after_pe/wh-w2n-zh-input.output.wav \
  --gen_seconds 8.58

The terminal also shows the detected task and target duration, and writes a compact JSON manifest under assets/after_pe/.

Python API

The snippets below show a few representative tasks; for the full instruction templates and per-task Python examples, see the Cookbook.

from auk.infer.infer_auk import AukInfer, save_audio

engine = AukInfer(
    "ckpts/AuK/config.yaml",
    "ckpts/AuK/auk_base.safetensors",
)

# Content Editing. Task with reference audio
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Replace 'but accepting what we cannot have' with 'and living well with dreams unmet'."},
            {"type": "audio", "audio": "assets/demo-input-audio/content-edit/content.wav"},
        ],
    }
]

audio, sample_rate = engine.generate(messages, gen_seconds=7.0)
save_audio(audio, sample_rate, "out_content_edit.wav")

# Instruct TTS. Task without reference audio
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "请基于下面的描述: \"一位二十多岁的女生，在恋人刚回家时，用温柔、体贴、关心且略带撒娇的语气说话。声音柔和亲近，语速稍慢，音量适中，音色清甜自然，句尾轻轻上扬，语调温暖，口齿清晰。\",生成语音内容\"欢迎回来宝宝,今天上班累不累呀\"."},
        ],
    }
]

audio, sr = engine.generate(messages, gen_seconds=4.0)
save_audio(audio, sr, "instruct.wav")

Fine-tuning

AuK provides a lightweight fine-tuning pipeline in src/auk/train/train.py. Training data is a JSONL file where each line is one source-target pair: the user message carries the instruction and optional source/reference audio, the assistant message carries the target audio, and duration (target seconds) drives dynamic batching.

{
  "duration": 5.66,
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Say the following with the same voice: \"你好，这是一次语音合成测试。\""},
        {"type": "audio", "audio_url": "/abs/path/ref.wav"}
      ]
    },
    {
      "role": "assistant",
      "content": [
        {"type": "audio", "audio_url": "/abs/path/target.wav"}
      ]
    }
  ]
}

All supported tasks use this same format — just change the instruction and the source/target audio (the audio field is omitted for instruction-only tasks). Set train_jsonl (and optionally val_jsonl) in scripts/train.sh, then run:

bash scripts/train.sh

📖 For the full guide — configuration reference, dynamic batching, checkpoints & resuming, EMA, monitoring, and troubleshooting — see docs/FINETUNING.md.

Contributing

Contributions are welcome — bug reports, documentation, tests, inference fixes, UI improvements, and performance work. Please read the Contributing Guide before opening an issue or pull request, and note that all participation is governed by our Code of Conduct.

Citation

If you find AuK useful in your research, please cite our work:

@misc{ma2026auktechnicalreportopensource,
  title         = {AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing},
  author        = {Ziyang Ma and Zhikang Niu and Wenming Tu and Tianrui Wang and Ruiqi Yan and Junxi Liu and Yanru Huo and Nickk Huang and Yang Liu and Qicong Xie and Zeyu Xie and Hui Wang and Haitao Li and Zixuan Jiang and Yalin Li and Jie Fang and Yifan Duan and Zeyue

AuK (Tencent-Hunyuan/AuK)

Searching Database…

1.5K ·GitHub
AuK
这是什么？腾讯混元开源的一个 15 亿参数基础模型，用于语音生成和编辑：零样本 TTS、指令式 TTS、语音编辑、增强和音源分离——全部通过同一个自然语言指令界面完成。
那它是安装的软件，还是在代码里调用的模型？某种意义上两者都是：你下载模型权重后，可以获得一个命令行工具（auk-infer）、一个交互式 Gradio 网页演示、ComfyUI 节点以及一个 Python API——四种方式对应同一批模型权重。
我怎么把它跑起来？克隆仓库，用 uv 或 Conda 创建一个隔离的 Python 3.10 环境并安装，从 Hugging Face 或 ModelScope 下载权重，然后运行 auk-infer 命令行工具。想用网页界面，需加装 gradio 扩展：pip install -e '.[gradio]'。
有没有我会碰到的硬性限制？ComfyUI 集成把单次运行限制在 30 秒的源音频加目标音频序列内。另外，GitHub 上显示的许可证是 NOASSERTION 而不是标准的 SPDX 许可证，所以商用前请先阅读自定义条款。
我的音频会留在本地电脑上吗？模型本身在你自己的硬件上本地运行。唯一的云端环节是可选的：对于基于音频的 Prompt Enhancer 任务，你可以接入腾讯云 ASR；如果没有配置它的凭据，AuK 会回退到本地的 SenseVoiceSmall 模型。
我的笔记本能跑吗，还是必须有 GPU？建议使用 CUDA GPU：默认精度是 bf16，推荐的 Gradio 配置会把 Base 和 Flash 两个模型分别加载到两块 GPU 上。整套方案还依赖一个 Qwen2.5-Omni-3B 编码器，无法替换为 Qwen3-Omni。
AuK 和 AuK-Flash 有什么区别？AuK 是以更高音质为目标调优的基础模型，NFE 和 CFG 均可配置。AuK-Flash 为速度做了蒸馏：固定 4 步且 CFG=0。在 Gradio 演示中，你可以只加载其中一个，也可以两个都加载，然后通过模型选择器切换。
它到底支持哪些语音任务？零样本 TTS（用参考音色朗读你的文字）、仅凭音色描述的指令式 TTS、内容与歌词改写、音调/语速/音量调整、情感、音色和去口音处理、耳语转换、降噪与增强、语音与音乐分离，以及目标说话人提取。
能在 ComfyUI 里用吗？可以：在 ComfyUI 所在的环境里安装 '.[comfyui]' 扩展，把 comfyui/ComfyUI-AuK 链接到 ComfyUI 的 custom_nodes 文件夹，然后打开自带的 auk.json 工作流——它提供了 AuK Model Loader 和 AuK Generate/Edit 节点，可用于生成、编辑、增强和分离。
我能用自己 的音频微调它吗？可以——src/auk/train/train.py 里有一个轻量级的训练流程。训练数据是 JSONL 格式，每行把一条指令消息（可附源音频或参考音频）与一段目标音频配对，且一种格式适用于所有任务。把 scripts/train.sh 指向你的文件即可；完整说明见 docs/FINETUNING.md。
What are similar projects?FireRedTTS3, DramaBox, OpenVoice, VoxCPM, index-tts, OmniVoice, dia, csm
Which topics is it in?zero-shot-tts, speech, foundation-models, diffusion, comfyui, speech-synthesis, gradio, voice, tts, text-to-speech, qwen, fine-tuning
Creator
Tencent-Hunyuan
Size
21.6 MB
Stars
1.5K
Updated
CommitOct 14
Age
2mo
Commits · 90 Days