Qwen3 README

The project's README from github.com/QwenLM/Qwen3. As one markdown file: README.md.

Qwen3

💜 Qwen Chat   |   🤗 Hugging Face   |   🤖 ModelScope   |    📑 Paper    |    📑 Blog    ｜   📖 Documentation
🖥️ Demo   |   💬 WeChat (微信)   |   🫨 Discord  

Visit our Hugging Face or ModelScope organization (click links above), search checkpoints with names starting with Qwen3- or visit the Qwen3 collection, and you will find all you need! Enjoy!

To learn more about Qwen3, feel free to read our documentation [EN|ZH]. Our documentation consists of the following sections:

Introduction

Qwen3-2507

Over the past three months, we continued to explore the potential of the Qwen3 families and we are excited to introduce the updated Qwen3-2507 in two variants, Qwen3-Instruct-2507 and Qwen3-Thinking-2507, and three sizes, 235B-A22B, 30B-A3B, and 4B.

Qwen3-Instruct-2507 is the updated version of the previous Qwen3 non-thinking mode, featuring the following key enhancements:

Qwen3-Thinking-2507 is the continuation of Qwen3 thinking model, with improved quality and depth of reasoning, featuring the following key enhancements:

Previous Qwen3 Release

Qwen3 (aka Qwen3-2504)

We are excited to announce the release of Qwen3, the latest addition to the Qwen family of large language models. These models represent our most advanced and intelligent systems to date, improving from our experience in building QwQ and Qwen2.5. We are making the weights of Qwen3 available to the public, including both dense and Mixture-of-Expert (MoE) models.

The highlights from Qwen3 include:

  • Dense and Mixture-of-Experts (MoE) models of various sizes, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B.
  • Seamless switching between thinking mode (for complex logical reasoning, math, and coding) and non-thinking mode (for efficient, general-purpose chat), ensuring optimal performance across various scenarios.
  • Significantly enhancement in reasoning capabilities, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning.
  • Superior human preference alignment, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience.
  • Expertise in agent capabilities, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks.
  • Support of 100+ languages and dialects with strong capabilities for multilingual instruction following and translation.

News

Performance

Detailed evaluation results are reported in this 📑 blog (Qwen3-2504) and this 📑 blog (Qwen3-2507) [coming soon].

For requirements on GPU memory and the respective throughput, see results here.

Run Qwen3

🤗 Transformers

Transformers is a library of pretrained natural language processing for inference and training. The latest version of transformers is recommended and transformers>=4.51.0 is required.

Qwen3-Instruct-2507

The following contains a code snippet illustrating how to use Qwen3-30B-A3B-Instruct-2507 to generate content based on given inputs. 

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-30B-A3B-Instruct-2507"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=16384
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

content = tokenizer.decode(output_ids, skip_special_tokens=True)

print("content:", content)

[!Note] Qwen3-Instruct-2507 supports only non-thinking mode and does not generate <think></think> blocks in its output. Meanwhile, specifying enable_thinking=False is no longer required.

Qwen3-Thinking-2507

The following contains a code snippet illustrating how to use Qwen3-30B-A3B-Thinking-2507 to generate content based on given inputs. 

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-30B-A3B-Thinking-2507"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print("thinking content:", thinking_content)  # no opening <think> tag
print("content:", content)

[!Note] Qwen3-Thinking-2507 supports only thinking mode. Additionally, to enforce model thinking, the default chat template automatically includes <think>. Therefore, it is normal for the model's output to contain only </think> without an explicit opening <think> tag.

Qwen3-Thinking-2507 also features an increased thinking length. We strongly recommend its use in highly complex reasoning tasks with adequate maximum generation length.

Switching Thinking/Non-thinking Modes for Previous Qwen3 Models

By default, Qwen3 models will think before response. This could be controlled by

  • enable_thinking=False: Passing enable_thinking=False to `tokenizer.apply_chat_template` will strictly prevent the model from generating thinking content.
  • /think and /no_think instructions: Use those words in the system or user message to signify whether Qwen3 should think. In multi-turn conversations, the latest instruction is followed.

ModelScope

We strongly advise users especially those in mainland China to use ModelScope. ModelScope adopts a Python API similar to Transformers. The CLI tool modelscope download can help you solve issues concerning downloading checkpoints. For vLLM and SGLang, the environment variable VLLM_USE_MODELSCOPE=true and SGLANG_USE_MODELSCOPE=true can be used respectively.

llama.cpp

llama.cpp enables LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware. llama.cpp>=b5401 is recommended for the full support of Qwen3.

To use the CLI, run the following in a terminal:

./llama-cli -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift
# CTRL+C to exit

To use the API server, run the following in a terminal:

./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift --port 8080

A simple web front end will be at http://localhost:8080 and an OpenAI-compatible API will be at http://localhost:8080/v1.

For additional guides, please refer to our documentation.

[!Note] llama.cpp adopts "rotating context management" and infinite generation is made possible by evicting earlier tokens. It could configured by parameters and the commands above effectively disable it. For more details, please refer to our documentation.

Ollama

After installing Ollama, you can initiate the Ollama service with the following command (Ollama v0.9.0 or higher is recommended):

ollama serve
# You need to keep this service running whenever you are using ollama

To pull a model checkpoint and run the model, use the ollama run command. You can specify a model size by adding a suffix to qwen3, such as :8b or :30b-a3b:

ollama run qwen3:8b
# Setting parameters, type "/set parameter num_ctx 40960" and "/set parameter num_predict 32768"
# To exit, type "/bye" and press ENTER
# For Qwen3-2504 models,
# - To enable thinking, which is the default, type "/set think"
# - To disable thinking, type "/set nothink"

You can also access the Ollama service via its OpenAI-compatible API. Please note that you need to (1) keep ollama serve running while using the API, and (2) execute ollama run qwen3:8b before utilizing this API to ensure that the model checkpoint is prepared. The API is at http://localhost:11434/v1/ by default.

For additional details, please visit ollama.ai.

[!Note] Ollama's naming may not be consistent with the Qwen's original naming. For example, qwen3:30b-a3b in Ollama points to qwen3:30b-a3b-thinking-2507-q4_K_M as of August 2025. Please check https://ollama.com/library/qwen3/tags before use.

[!Note] Ollama adopts the same "rotating context management" with llama.cpp. However, its default settings (num_ctx 2048 and num_predict -1), suggesting infinite generation with a 2048-token context, could lead to trouble for Qwen3 models. We recommend setting num_ctx and num_predict properly.

LMStudio

Qwen3 has already been supported by lmstudio.ai. You can directly use LMStudio with our GGUF files.

ExecuTorch

To export and run on ExecuTorch (iOS, Android, Mac, Linux, and more), please follow this example.

MNN

To export and run on MNN, which supports Qwen3 on mobile devices, please visit Alibaba MNN.

MLX LM

If you are running on Apple Silicon, mlx-lm also supports Qwen3 (mlx-lm>=0.24.0). Look for models ending with MLX on Hugging Face Hub.

OpenVINO

If you are running on Intel CPU or GPU, OpenVINO toolkit supports Qwen3. You can follow this chatbot example.

Deploy Qwen3

Qwen3 is supported by multiple inference frameworks. Here we demonstrate the usage of SGLang, vLLM and TensorRT-LLM. You can also find Qwen3 models from various inference providers, e.g., Alibaba Cloud Model Studio.

SGLang

SGLang is a fast serving framework for large language models and vision language models. SGLang could be used to launch a server with OpenAI-compatible API service. sglang>=0.4.6.post1 is required.

For Qwen3-Instruct-2507, 

python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 --port 30000 --context-length 262144

For Qwen3-Thinking-2507,

python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Thinking-2507 --port 30000 --context-length 262144 --reasoning-parser deepseek-r1

For Qwen3, it is

python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 30000 --context-length 131072 --reasoning-parser qwen3

An OpenAI-compatible API will be available at http://localhost:30000/v1.

[!Note] Due to the preprocessing of API requests in SGLang, which drops all reasoning_content fields, the quality of multi-step tool use with Qwen3 thinking models may be suboptimal, which requires the existence of the related thinking content. While the fixes are being worked on, as a workdaround, we recommend passing the content as it is, without extracting thinking content, and the chat template will correctly handle the processing.

vLLM

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. vllm>=0.9.0 is recommended.

For Qwen3-Instruct-2507, 

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 --port 8000 --max-model-len 262144

For Qwen3-Thinking-2507,

vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507 --port 8000 --max-model-len 262144 --enable-reasoning --reasoning-parser deepseek_r1

For Qwen3, it is

vllm serve Qwen/Qwen3-8B --port 8000 --max-model-len 131072 --enable-reasoning --reasoning-parser qwen3

An OpenAI-compatible API will be available at http://localhost:8000/v1.

[!Note] Due to the preprocessing of API requests in vLLM, which drops all reasoning_content fields, the quality of multi-step tool use with Qwen3 thinking models may be suboptimal, which requires the existence of the related thinking content. While the fixes are being worked on, as a workdaround, we recommend passing the content as it is, without extracting thinking content, and the chat template will correctly handle the processing.

TensorRT-LLM

TensorRT-LLM is an open-source LLM inference engine from NVIDIA, which provides optimizations including custom attention kernels, quantization and more on NVIDIA GPUs. Qwen3 is supported in its re-architected PyTorch backend. tensorrt_llm>=0.20.0rc3 is recommended. Please refer to the README page for more details.

trtllm-serve Qwen/Qwen3-8B --host localhost -

Qwen3 (QwenLM/Qwen3)

llamacpp
Searching Database…

27.7K ·GitHub
Qwen3
What is this?Qwen3 is a series of open-weight large language models built by the Qwen team at Alibaba Cloud, released as both dense and Mixture-of-Experts (MoE) models from 0.6B up to 235B-A22B parameters, with switchable 'thinking' and 'non-thinking' reasoning modes.
Is this a framework or just downloadable model weights?It is a collection of pretrained model checkpoints hosted on Hugging Face and ModelScope, not a framework itself — you pick a checkpoint and run it with separate tools like Transformers, vLLM, or llama.cpp.
How do I run Qwen3 locally?You can load it with Hugging Face Transformers (>=4.51.0) or ModelScope, or run GGUF builds locally with llama.cpp, Ollama, or LM Studio on CPU/GPU.
Which serving frameworks does it support for production deployment?For larger-scale, OpenAI-compatible serving it supports SGLang, vLLM (>=0.9.0), and TensorRT-LLM.
Any pitfalls when running the thinking models?llama.cpp and Ollama use 'rotating context management' that silently evicts old tokens, and Ollama's default 2048-token context can cut off long reasoning traces, so context limits need manual tuning; also vLLM and SGLang currently strip reasoning_content fields from requests, which can weaken multi-step tool-use quality.
Can Qwen3 run on mobile devices or Apple Silicon?Yes — it can be exported via ExecuTorch for iOS, Android, Mac, and Linux, runs on mobile through Alibaba's MNN, and is natively supported on Apple Silicon via MLX LM.
What's the difference between thinking and non-thinking mode?Thinking mode makes the model reason step by step before answering, which helps with complex math, logic, and coding; non-thinking mode answers directly for fast general chat. You can switch via enable_thinking or the /think and /nothink commands.
What sizes does Qwen3 come in?Dense models at 0.6B, 1.7B, 4B, 8B, 14B, and 32B, plus MoE models at 30B-A3B and 235B-A22B, so you can match a size to your hardware.
How long a context can Qwen3 handle?The updated Qwen3-2507 models support 256K-token context natively, extendable up to 1 million tokens for ultra-long inputs.
Why does the README recommend ModelScope?It's especially recommended for users in mainland China, offering a Transformers-like Python API and a CLI (modelscope download) for fetching checkpoints.
What are similar projects?Qwen3.8-Flash-Next, Hy-MT2, Qwen3.8, gpt-oss, MiniCPM, Hy4-preview, airllm, DeepSeek-V3
Which topics is it in?alibaba, qwen3, llamacpp, foundation-models, mixture-of-experts, reasoning, moe, gguf, transformers, vllm, qwen, quantization
Creator
QwenLM
Size
6.8 MB
Stars
27.7K
Updated
CommitJan 09
Age
2y
Commits · 90 Days