Qwen3-VL README

The project's README from github.com/QwenLM/Qwen3-VL. As one markdown file: README.md.

Qwen3-VL

💜 Qwen Chat   |   🤗 Hugging Face   |   🤖 ModelScope   |   📑 Blog   |   📚 Cookbooks   |   📑 Paper  
🖥️ Demo   |   💬 WeChat (微信)   |   🫨 Discord   |   📑 API   |   🖥️ PAI-DSW

Introduction

Meet Qwen3-VL — the most powerful vision-language model in the Qwen series to date.

This generation delivers comprehensive upgrades across the board: superior text understanding & generation, deeper visual perception & reasoning, extended context length, enhanced spatial and video dynamics comprehension, and stronger agent interaction capabilities.

Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning‑enhanced Thinking editions for flexible, on‑demand deployment.

Key Enhancements:

Model Architecture Updates:

  1. Interleaved-MRoPE: Full‑frequency allocation over time, width, and height via robust positional embeddings, enhancing long‑horizon video reasoning.

  2. DeepStack: Fuses multi‑level ViT features to capture fine‑grained details and sharpen image–text alignment.

  3. Text–Timestamp Alignment: Moves beyond T‑RoPE to precise, timestamp‑grounded event localization for stronger video temporal modeling.

News

Performance

Visual Tasks

Text-Centric Tasks

Cookbooks

We are preparing cookbooks for many capabilities, including recognition, localization, document parsing, video understanding, key information extraction, and more. Welcome to learn more!

Cookbook Description Open
Omni Recognition Not only identify animals, plants, people, and scenic spots but also recognize various objects such as cars and merchandise. Colab
Powerful Document Parsing Capabilities The parsing of documents has reached a higher level, including not only text but also layout position information and our Qwen HTML format. Colab
Precise Object Grounding Across Formats Using relative position coordinates, it supports both boxes and points, allowing for diverse combinations of positioning and labeling tasks. Colab
General OCR and Key Information Extraction Stronger text recognition capabilities in natural scenes and multiple languages, supporting diverse key information extraction needs. Colab
Video Understanding Better video OCR, long video understanding, and video grounding. Colab
Mobile Agent Locate and think for mobile phone control. Colab
Computer-Use Agent Locate and think for controlling computers and Web. Colab
3D Grounding Provide accurate 3D bounding boxes for both indoor and outdoor objects. Colab
Thinking with Images Utilize image_zoom_in_tool and search_tool to facilitate the model’s precise comprehension of fine-grained visual details within images. Colab
MultiModal Coding Generate accurate code based on rigorous comprehension of multimodal information. Colab
Long Document Understanding Achieve rigorous semantic comprehension of ultra-long documents. Colab
Spatial Understanding See, understand and reason about the spatial information Colab

Quickstart

Below, we provide simple examples to show how to use Qwen3-VL with 🤖 ModelScope and 🤗 Transformers.

# The Qwen3-VL model requires transformers >= 4.57.0
pip install "transformers>=4.57.0"

🤖 ModelScope

We strongly advise users especially those in mainland China to use ModelScope. snapshot_download can help you solve issues concerning downloading checkpoints.

Using 🤗 Transformers to Chat

Here we show a code snippet to show you how to use the chat model with transformers:

from transformers import AutoModelForImageTextToText, AutoProcessor

# default: Load the model on the available device(s)
model = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-235B-A22B-Instruct", dtype="auto", device_map="auto"
)

# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
# model = AutoModelForImageTextToText.from_pretrained(
#     "Qwen/Qwen3-VL-235B-A22B-Instruct",
#     dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-235B-A22B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

# Preparation for inference
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
)
inputs = inputs.to(model.device)

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
Multi image inference 
# Messages containing multiple images and a text query
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "file:///path/to/image1.jpg"},
            {"type": "image", "image": "file:///path/to/image2.jpg"},
            {"type": "text", "text": "Identify the similarities between these images."},
        ],
    }
]

# Preparation for inference
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
)
inputs = inputs.to(model.device)

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
Video inference 
# Messages containing a video url(or a local path) and a text query
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-VL/space_woaudio.mp4",
            },
            {"type": "text", "text": "Describe this video."},
        ],
    }
]

# Preparation for inference
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
)
inputs = inputs.to(model.device)

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
Batch inference 
# for batch generation, padding_side should be set to left!
processor.tokenizer.padding_side = 'left'

# Sample messages for batch inference
messages1 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "file:///path/to/image1.jpg"},
            {"type": "image", "image": "file:///path/to/image2.jpg"},
            {"type": "text", "text": "What are the common elements in these pictures?"},
        ],
    }
]
messages2 = [
    {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
    {"role": "user", "content": [{"type": "text", "text": "Who are you?"}]},
]
# Combine messages for batch processing
messages = [messages1, messages2]

# Preparation for inference
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
    padding=True # padding should be set for batch generation!
)
inputs = inputs.to(model.device)

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output

Qwen3-VL (QwenLM/Qwen3-VL)

Searching Database…

20.1K ·GitHub
Qwen3-VL
这是什么？Qwen3-VL 是由 Alibaba Cloud 的 Qwen 团队开发的一系列多模态视觉语言模型，兼具出色的文本理解能力，以及深入的视觉感知、推理和智能体能力。
这是单个模型，还是一整套不同规格的模型？这是一个包含多种规格的系列，涵盖 Dense 和 MoE 两种架构，从适合边缘设备的小尺寸到云端的大规模模型都有，每种规格都提供直接回答的 Instruct 版本和增强推理的 Thinking 版本。
我该怎么开始使用？你可以用 Hugging Face Transformers（AutoProcessor/AutoModel）加载模型，并使用提供的代码示例与其对话；也可以通过 ModelScope 下载模型权重，团队特别推荐中国大陆用户使用这种方式。
所有规格的模型都同样容易运行吗？不是。该系列从 2B 的 Dense 模型一直到 235B-A22B 的 MoE 旗舰模型，所需算力差异很大；部分较大的模型提供 AWQ/GPTQ 量化版本，但仍然需要较大的 GPU 显存。
运行较大的模型是不是一定需要高性能 GPU？较大的模型确实需要。项目提供了量化版本（AWQ、GPTQ），让最耗资源的模型更容易运行，但即便如此，70B+ 及 MoE 版本仍需要多 GPU 或大显存的配置。
Instruct 版和 Thinking 版有什么区别？Instruct 版本会直接、即时地给出回答，Thinking 版本则会加入明确的推理过程，用于处理更难的视觉和逻辑任务，类似于早期专注视觉推理的研究模型 QvQ-72B-Preview。
Visual Agent 功能具体能让模型做什么？它能让模型直接操作电脑和手机的图形界面：识别屏幕上的元素、理解其功能、调用工具，并自主完成多步骤任务。
它能处理多长的上下文和视频？Qwen3-VL 原生支持 256K 的上下文窗口，可扩展至 1M 个 token，能够处理整本书或数小时的视频，并完整回溯内容、实现秒级事件索引。
和早期的 Qwen 模型相比，它的 OCR 能力怎么样？它支持 32 种语言，此前为 10 种，并且在低光照、模糊、倾斜、生僻字和古文字以及复杂的长文档版式下，都更加稳健。
What are similar projects?ZDTaichu5.0-9B, MiniCPM, OraRL, Kimi-K3, Valen, Qwen3.8-Flash-Next, airllm, MiniCPM-V
Which topics is it in?video-understanding, alibaba, edge-ai, vision-language-model, reasoning, moe, ocr, document-processing, qwen, jupyter notebook, quantization, multimodal
Creator
QwenLM
Size
134.8 MB
Stars
20.1K
Updated
CommitJan 30
Age
2y
Commits · 90 Days