Qwen3-Omni README

The project's README from github.com/QwenLM/Qwen3-Omni. As one markdown file: README.md.

Qwen3-Omni


💜 Qwen Chat   |   🤗 Hugging Face   |   🤖 ModelScope   |   📑 Blog   |   📚 Cookbooks   |   📑 Paper  
🖥️ Hugging Face Demo   |    🖥️ ModelScope Demo   |   💬 WeChat (微信)   |   🫨 Discord   |   📑 API

We release Qwen3-Omni, the natively end-to-end multilingual omni-modal foundation models. It is designed to process diverse inputs including text, images, audio, and video, while delivering real-time streaming responses in both text and natural speech. Click the video below for more information 😃

English Version Open English Video
Chinese Version 打开中文视频

News

Contents

Overview

Introduction

Qwen3-Omni is the natively end-to-end multilingual omni-modal foundation models. It processes text, images, audio, and video, and delivers real-time streaming responses in both text and natural speech. We introduce several architectural upgrades to improve performance and efficiency. Key features:

Model Architecture

Cookbooks for Usage Cases

Qwen3-Omni supports a wide range of multimodal application scenarios, covering various domain tasks involving audio, image, video, and audio-visual modalities. Below are several cookbooks demonstrating the usage cases of Qwen3-Omni and these cookbooks include our actual execution logs. You can first follow the QuickStart guide to download the model and install the necessary inference environment dependencies, then run and experiment locally—try modifying prompts or switching model types, and enjoy exploring the capabilities of Qwen3-Omni!

Category Cookbook Description Open
Audio Speech Recognition Speech recognition, supporting multiple languages and long audio. Open In Colab
Speech Translation Speech-to-Text / Speech-to-Speech translation. Open In Colab
Music Analysis Detailed analysis and appreciation of any music, including style, genre, rhythm, etc. Open In Colab
Sound Analysis Description and analysis of various sound effects and audio signals. Open In Colab
Audio Caption Audio captioning, detailed description of any audio input. Open In Colab
Mixed Audio Analysis Analysis of mixed audio content, such as speech, music, and environmental sounds. Open In Colab
Visual OCR OCR for complex images. Open In Colab
Object Grounding Target detection and grounding. Open In Colab
Image Question Answering arbitrary questions about any image. Open In Colab
Image Math Solving complex mathematical problems in images, highlighting the capabilities of the Thinking model. Open In Colab
Video Description Detailed description of video content. Open In Colab
Video Navigation Generating navigation commands from first-person motion videos. Open In Colab
Video Scene Transition Analysis of scene transitions in videos. Open In Colab
Audio-Visual Audio Visual Question Answering arbitrary questions in audio-visual scenarios, demonstrating the model's ability to model temporal alignment between audio and video. Open In Colab
Audio Visual Interaction Interactive communication with the model using audio-visual inputs, including task specification via audio. Open In Colab
Audio Visual Dialogue Conversational interaction with the model using audio-visual inputs, showcasing its capabilities in casual chat and assistant-like behavior. Open In Colab
Agent Audio Function Call Using audio input to perform function calls, enabling agent-like behaviors. Open In Colab
Downstream Task Fine-tuning Omni Captioner Introduction and capability demonstration of Qwen3-Omni-30B-A3B-Captioner, a downstream fine-tuned model based on Qwen3-Omni-30B-A3B-Instruct, illustrating the strong generalization ability of the Qwen3-Omni foundation model. Open In Colab

QuickStart

Here, we provide several methods to quickly get started with Qwen3-Omni. If you want complete experience of Qwen3-Omni, you can use Hugging Face Transformers. However, since Qwen3-Omni employs an MoE architecture, inference speed with Hugging Face Transformers on MoE models can be very slow. For large-scale invocation or low-latency requirements, we highly recommend using vLLM or performing inference via the DashScope API. We also strongly suggest using our provided Docker image, which includes a complete runtime environment for both Hugging Face Transformers and vLLM. In addition, our cookbooks offer some use cases to show Qwen3-Omni's capabilities. Welcome to learn more!

Model Description and Download

Below is the description of all Qwen3-Omni models. Please select and download the model that fits your needs.

Model Name Description
Qwen3-Omni-30B-A3B-Instruct The Instruct model of Qwen3-Omni-30B-A3B, containing both thinker and talker, supporting audio, video, and text input, with audio and text output. For more information, please read the Qwen3-Omni Technical Report.
Qwen3-Omni-30B-A3B-Thinking The Thinking model of Qwen3-Omni-30B-A3B, containing the thinker component, equipped with chain-of-thought reasoning, supporting audio, video, and text input, with text output. For more information, please read the Qwen3-Omni Technical Report.
Qwen3-Omni-30B-A3B-Captioner A downstream audio fine-grained caption model fine-tuned from Qwen3-Omni-30B-A3B-Instruct, which produces detailed, low-hallucination captions for arbitrary audio inputs. It contains the thinker, supporting audio input and text output. For more information, you can refer to the model's cookbook or Hugging Face Demo and ModelScope Demo.

During loading in Hugging Face Transformers or vLLM, model weights will be automatically downloaded based on the model name. However, if your runtime environment is not conducive to downloading weights during execution, you can refer to the following commands to manually download the model weights to a local directory:

# Download through ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Instruct --local_dir ./Qwen3-Omni-30B-A3B-Instruct
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Thinking --local_dir ./Qwen3-Omni-30B-A3B-Thinking
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Captioner --local_dir ./Qwen3-Omni-30B-A3B-Captioner

# Download through Hugging Face
pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Instruct --local-dir ./Qwen3-Omni-30B-A3B-Instruct
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Thinking --local-dir ./Qwen3-Omni-30B-A3B-Thinking
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Captioner --local-dir ./Qwen3-Omni-30B-A3B-Captioner

Transformers Usage

Installation

We recommend using Transformers version 5.2.0 or later for the best performance and accuracy. Older versions, such as Transformers 4.57.x, not only deliver weaker performance, but also provide lower accuracy compared with versions 5.2.0 and above.

# If you already have transformers installed, please uninstall it first, or create a new Python environment
# pip uninstall transformers
pip install "transformers>=5.2.0"
pip install accelerate

We offer a toolkit to help you handle various types of audio and visual input more conveniently, providing an API-like experience. This includes support for base64, URLs, and interleaved audio, images, and videos. You can install it using the following command and make sure your system has ffmpeg installed:

pip install qwen-omni-utils -U

Additionally, we recommend using FlashAttention 2 when running with Hugging Face Transformers to reduce GPU memory usage. However, if you are primarily using vLLM for inference, this installation is not necessary, as vLLM includes FlashAttention 2 by default.

pip install -U flash-attn --no-build-isolation

Also, you should have hardware that is compatible with FlashAttention 2. Read more about it in the official documentation of the FlashAttention repository. FlashAttention 2 can only be used when a model is loaded in torch.float16 or torch.bfloat16.

Code Snippet

Here is a code snippet to show you how to use Qwen3-Omni with transformers and qwen_omni_utils:

import soundfile as sf

from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from qwen_omni_utils import process_mm_info

MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
# MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"

Qwen3-Omni (QwenLM/Qwen3-Omni)

Searching Database…

4.1K ·GitHub
Qwen3-Omni
这是什么？Qwen3-Omni 是阿里 Qwen 团队推出的开放式原生端到端全模态大语言模型系列：它可以接收文本、图片、音频和视频作为输入，并以文本或自然语音的形式实时流式输出回答。
这是我自己下载运行的东西，还是在线服务？两者都是。你可以从 Hugging Face 下载开放权重，用 Transformers 或 vLLM 自己运行模型，也可以不用自己准备硬件，直接通过阿里的 DashScope API 调用。
我怎么把它跑起来？推荐的方式是使用官方提供的 Docker 镜像，它内置了 Transformers 和 vLLM 两套完整的运行环境；之后只需加载诸如 Qwen/Qwen3-Omni-30B-A3B-Instruct 这样的检查点，权重就会自动下载。
我需要先装好哪些东西？需要 Transformers 5.2.0 或更高版本——像 4.57.x 这样的旧版本效果较差、准确度更低——还需要 qwen-omni-utils 工具包来处理 base64、URL 以及交错的音频、图片、视频输入，而它要求系统里装有 ffmpeg。
在本地运行它有什么坑？主要是 Transformers 这条路径的速度：由于模型是 Mixture-of-Experts（专家混合）结构，直接用 Hugging Face Transformers 推理可能非常慢，所以作者建议在低延迟或大规模场景下改用 vLLM。
没有 GPU 能跑吗？基本不行：这是一个 300 亿参数的 MoE 模型，默认就是靠 GPU 推理的。用 Transformers 时，README 建议启用 FlashAttention 2（仅支持 fp16 或 bf16）来降低显存占用，而 vLLM 默认就包含 FlashAttention 2。
它能听懂和说出哪些语言？文本支持 119 种语言。语音输入支持 19 种语言——包括英语、中文、日语、韩语、德语、俄语、法语、西班牙语、粤语、阿拉伯语等——语音输出支持 10 种，包括英语、中文、法语、德语、俄语、意大利语、西班牙语、葡萄牙语、日语和韩语。
和 Gemini 这类闭源模型比，它的表现怎么样？在 36 项音频和视频基准测试中，它在 22 项上达到了业界最先进水平，在 32 项上达到了开源模型中的最先进水平；其语音识别、音频理解和语音对话能力可与 Gemini 2.5 Pro 相媲美。发布几天后，它还登上了 Hugging Face Trending 榜单第一。
Instruct、Thinking 和 Captioner 这几个版本有什么区别？三者都是 30B-A3B 的 MoE 模型。Instruct 同时包含 thinker（思考）和 talker（说话）两个组件，可以用音频和文本回答；Thinking 增加了思维链推理能力，但只能用文本回答；Captioner 则经过专门微调，能为任意音频输入生成详细且低幻觉的描述。
这些示例教程都涵盖哪些任务？包括语音识别、语音转文字或语音到语音的翻译、音乐和音效分析、音频内容描述、图片 OCR 和图片中的数学题、视频描述以及基于第一人称视频的导航、视听对话，还有用于实现类智能体行为的语音驱动函数调用——每项都附有实际的运行日志。
What are similar projects?Qwen3-ASR, Speech, Hy-MT2, seamless_communication, WhisperLiveKit, verl, VibeVoice, airllm
Which topics is it in?conversational-ai, moe, speech-to-text, transformers, speech-recognition, vllm, text-to-speech, translation, qwen, jupyter notebook, multimodal, large-language-models
Creator
QwenLM
Size
25.5 MB
Stars
4.1K
Updated
CommitApr 23
Age
1y
Commits · 90 Days