Qwen3-Omni README

The project's README from github.com/QwenLM/Qwen3-Omni. As one markdown file: README.md.

Qwen3-Omni


💜 Qwen Chat   |   🤗 Hugging Face   |   🤖 ModelScope   |   📑 Blog   |   📚 Cookbooks   |   📑 Paper  
🖥️ Hugging Face Demo   |    🖥️ ModelScope Demo   |   💬 WeChat (微信)   |   🫨 Discord   |   📑 API

We release Qwen3-Omni, the natively end-to-end multilingual omni-modal foundation models. It is designed to process diverse inputs including text, images, audio, and video, while delivering real-time streaming responses in both text and natural speech. Click the video below for more information 😃

English Version Open English Video
Chinese Version 打开中文视频

News

Contents

Overview

Introduction

Qwen3-Omni is the natively end-to-end multilingual omni-modal foundation models. It processes text, images, audio, and video, and delivers real-time streaming responses in both text and natural speech. We introduce several architectural upgrades to improve performance and efficiency. Key features:

Model Architecture

Cookbooks for Usage Cases

Qwen3-Omni supports a wide range of multimodal application scenarios, covering various domain tasks involving audio, image, video, and audio-visual modalities. Below are several cookbooks demonstrating the usage cases of Qwen3-Omni and these cookbooks include our actual execution logs. You can first follow the QuickStart guide to download the model and install the necessary inference environment dependencies, then run and experiment locally—try modifying prompts or switching model types, and enjoy exploring the capabilities of Qwen3-Omni!

Category Cookbook Description Open
Audio Speech Recognition Speech recognition, supporting multiple languages and long audio. Open In Colab
Speech Translation Speech-to-Text / Speech-to-Speech translation. Open In Colab
Music Analysis Detailed analysis and appreciation of any music, including style, genre, rhythm, etc. Open In Colab
Sound Analysis Description and analysis of various sound effects and audio signals. Open In Colab
Audio Caption Audio captioning, detailed description of any audio input. Open In Colab
Mixed Audio Analysis Analysis of mixed audio content, such as speech, music, and environmental sounds. Open In Colab
Visual OCR OCR for complex images. Open In Colab
Object Grounding Target detection and grounding. Open In Colab
Image Question Answering arbitrary questions about any image. Open In Colab
Image Math Solving complex mathematical problems in images, highlighting the capabilities of the Thinking model. Open In Colab
Video Description Detailed description of video content. Open In Colab
Video Navigation Generating navigation commands from first-person motion videos. Open In Colab
Video Scene Transition Analysis of scene transitions in videos. Open In Colab
Audio-Visual Audio Visual Question Answering arbitrary questions in audio-visual scenarios, demonstrating the model's ability to model temporal alignment between audio and video. Open In Colab
Audio Visual Interaction Interactive communication with the model using audio-visual inputs, including task specification via audio. Open In Colab
Audio Visual Dialogue Conversational interaction with the model using audio-visual inputs, showcasing its capabilities in casual chat and assistant-like behavior. Open In Colab
Agent Audio Function Call Using audio input to perform function calls, enabling agent-like behaviors. Open In Colab
Downstream Task Fine-tuning Omni Captioner Introduction and capability demonstration of Qwen3-Omni-30B-A3B-Captioner, a downstream fine-tuned model based on Qwen3-Omni-30B-A3B-Instruct, illustrating the strong generalization ability of the Qwen3-Omni foundation model. Open In Colab

QuickStart

Here, we provide several methods to quickly get started with Qwen3-Omni. If you want complete experience of Qwen3-Omni, you can use Hugging Face Transformers. However, since Qwen3-Omni employs an MoE architecture, inference speed with Hugging Face Transformers on MoE models can be very slow. For large-scale invocation or low-latency requirements, we highly recommend using vLLM or performing inference via the DashScope API. We also strongly suggest using our provided Docker image, which includes a complete runtime environment for both Hugging Face Transformers and vLLM. In addition, our cookbooks offer some use cases to show Qwen3-Omni's capabilities. Welcome to learn more!

Model Description and Download

Below is the description of all Qwen3-Omni models. Please select and download the model that fits your needs.

Model Name Description
Qwen3-Omni-30B-A3B-Instruct The Instruct model of Qwen3-Omni-30B-A3B, containing both thinker and talker, supporting audio, video, and text input, with audio and text output. For more information, please read the Qwen3-Omni Technical Report.
Qwen3-Omni-30B-A3B-Thinking The Thinking model of Qwen3-Omni-30B-A3B, containing the thinker component, equipped with chain-of-thought reasoning, supporting audio, video, and text input, with text output. For more information, please read the Qwen3-Omni Technical Report.
Qwen3-Omni-30B-A3B-Captioner A downstream audio fine-grained caption model fine-tuned from Qwen3-Omni-30B-A3B-Instruct, which produces detailed, low-hallucination captions for arbitrary audio inputs. It contains the thinker, supporting audio input and text output. For more information, you can refer to the model's cookbook or Hugging Face Demo and ModelScope Demo.

During loading in Hugging Face Transformers or vLLM, model weights will be automatically downloaded based on the model name. However, if your runtime environment is not conducive to downloading weights during execution, you can refer to the following commands to manually download the model weights to a local directory:

# Download through ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Instruct --local_dir ./Qwen3-Omni-30B-A3B-Instruct
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Thinking --local_dir ./Qwen3-Omni-30B-A3B-Thinking
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Captioner --local_dir ./Qwen3-Omni-30B-A3B-Captioner

# Download through Hugging Face
pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Instruct --local-dir ./Qwen3-Omni-30B-A3B-Instruct
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Thinking --local-dir ./Qwen3-Omni-30B-A3B-Thinking
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Captioner --local-dir ./Qwen3-Omni-30B-A3B-Captioner

Transformers Usage

Installation

We recommend using Transformers version 5.2.0 or later for the best performance and accuracy. Older versions, such as Transformers 4.57.x, not only deliver weaker performance, but also provide lower accuracy compared with versions 5.2.0 and above.

# If you already have transformers installed, please uninstall it first, or create a new Python environment
# pip uninstall transformers
pip install "transformers>=5.2.0"
pip install accelerate

We offer a toolkit to help you handle various types of audio and visual input more conveniently, providing an API-like experience. This includes support for base64, URLs, and interleaved audio, images, and videos. You can install it using the following command and make sure your system has ffmpeg installed:

pip install qwen-omni-utils -U

Additionally, we recommend using FlashAttention 2 when running with Hugging Face Transformers to reduce GPU memory usage. However, if you are primarily using vLLM for inference, this installation is not necessary, as vLLM includes FlashAttention 2 by default.

pip install -U flash-attn --no-build-isolation

Also, you should have hardware that is compatible with FlashAttention 2. Read more about it in the official documentation of the FlashAttention repository. FlashAttention 2 can only be used when a model is loaded in torch.float16 or torch.bfloat16.

Code Snippet

Here is a code snippet to show you how to use Qwen3-Omni with transformers and qwen_omni_utils:

import soundfile as sf

from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from qwen_omni_utils import process_mm_info

MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
# MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"

Qwen3-Omni (QwenLM/Qwen3-Omni)

Searching Database…

4.1K ·GitHub
Qwen3-Omni
Что это такое?Qwen3-Omni — это семейство открытых омнимодальных LLM от команды Qwen из Alibaba, изначально сквозных (end-to-end): они принимают на вход текст, изображения, аудио и видео и в реальном времени отдают ответ текстом или естественной речью.
Это то, что я скачиваю и запускаю сам, или онлайн-сервис?И то и другое. Можно скачать открытые веса с Hugging Face и запустить модель самостоятельно через Transformers или vLLM, а можно обойтись без железа и обращаться к ней через API Alibaba DashScope.
Как его запустить?Рекомендуемый путь — готовый Docker-образ, в который уже включена полная среда для Transformers и vLLM; после этого остаётся загрузить чекпоинт, например Qwen/Qwen3-Omni-30B-A3B-Instruct, и веса скачаются автоматически.
Что нужно установить заранее?Transformers 5.2.0 или новее — старые версии вроде 4.57.x работают хуже и менее точно — плюс набор инструментов qwen-omni-utils для обработки base64, URL и чередующихся аудио-, видео- и изображений-входов, а ему нужен установленный в системе ffmpeg.
В чём подвох локального запуска?Скорость на пути через Transformers: поскольку модель построена на Mixture-of-Experts, обычный инференс через Hugging Face Transformers может быть очень медленным, поэтому авторы советуют vLLM для сценариев с низкой задержкой или большим масштабом.
А без видеокарты он заработает?По сути нет: это MoE-модель на 30B параметров, так что подразумевается инференс на GPU. В Transformers в README рекомендуется FlashAttention 2 (только fp16 или bf16), чтобы снизить потребление видеопамяти, а в vLLM FlashAttention 2 включён по умолчанию.
Какие языки он понимает и на каких говорит?Текст — на 119 языках. Речь на вход он принимает на 19 языках — английский, китайский, японский, корейский, немецкий, русский, французский, испанский, кантонский, арабский и другие — и отвечает голосом на 10, включая английский, китайский, французский, немецкий, русский, итальянский, испанский, португальский, японский и корейский.
Насколько он хорош по сравнению с закрытыми моделями вроде Gemini?Он показывает лучший результат на 22 из 36 аудио- и видеобенчмарков и лучший среди open-source на 32 из 36, при этом по распознаванию речи, пониманию аудио и голосовому диалогу он сравним с Gemini 2.5 Pro. Через несколько дней после релиза он также поднялся на первое место в Hugging Face Trending.
Чем отличаются варианты Instruct, Thinking и Captioner?Все три — MoE-модели 30B-A3B. В Instruct есть и блок thinker, и блок talker, и он отвечает и аудио, и текстом; Thinking добавляет цепочку рассуждений (chain-of-thought), но отвечает только текстом; Captioner дообучен выдавать подробные описания с низкой галлюцинацией для произвольного аудиовхода.
Какие задачи разобраны в cookbook-примерах?Распознавание речи и перевод речи в текст или речи в речь, анализ музыки и звуковых эффектов, описание аудио, OCR на изображениях и решение математических задач по картинкам, описание видео и навигация по съёмке от первого лица, аудиовизуальный чат и вызов функций голосом для поведения в духе агента — всё с реальными логами выполнения.
What are similar projects?Qwen3-ASR, Speech, Hy-MT2, seamless_communication, WhisperLiveKit, verl, VibeVoice, airllm
Which topics is it in?conversational-ai, moe, speech-to-text, transformers, speech-recognition, vllm, text-to-speech, translation, qwen, jupyter notebook, multimodal, large-language-models
Creator
QwenLM
Size
25.5 MB
Stars
4.1K
Updated
CommitApr 23
Age
1y
Commits · 90 Days