Qwen README

The project's README from github.com/QwenLM/Qwen. As one markdown file: README.md.

中文  ｜  English  ｜  日本語 ｜  Français ｜  Español




🤗 Hugging Face   |   🤖 ModelScope   |    📑 Paper    ｜   🖥️ Demo
WeChat (微信)   |   Discord   ｜   API



[!Important] Qwen2 is here! You are welcome to follow QwenLM/Qwen2 and share your experience there.

This repo (QwenLM/Qwen) is no longer actively maintained, due to substantial codebase differences.


Qwen-Chat Qwen-Chat (Int4) Qwen-Chat (Int8) Qwen
1.8B 🤖 🤗 🤖 🤗 🤖 🤗 🤖 🤗
7B 🤖 🤗 🤖 🤗 🤖 🤗 🤖 🤗
14B 🤖 🤗 🤖 🤗 🤖 🤗 🤖 🤗
72B 🤖 🤗 🤖 🤗 🤖 🤗 🤖 🤗

We opensource our Qwen series, now including Qwen, the base language models, namely Qwen-1.8B, Qwen-7B, Qwen-14B, and Qwen-72B, as well as Qwen-Chat, the chat models, namely Qwen-1.8B-Chat, Qwen-7B-Chat, Qwen-14B-Chat, and Qwen-72B-Chat. Links are on the above table. Click them and check the model cards. Also, we release the technical report. Please click the paper link and check it out!

In brief, we have strong base language models, which have been stably pretrained for up to 3 trillion tokens of multilingual data with a wide coverage of domains, languages (with a focus on Chinese and English), etc. They are able to achieve competitive performance on benchmark datasets. Additionally, we have chat models that are aligned with human preference based on SFT and RLHF (not released yet), which are able to chat, create content, extract information, summarize, translate, code, solve math problems, and so on, and are able to use tools, play as agents, or even play as code interpreters, etc.

Model Release Date Max Length System Prompt Enhancement # of Pretrained Tokens Minimum GPU Memory Usage of Finetuning (Q-Lora) Minimum GPU Usage of Generating 2048 Tokens (Int4) Tool Usage
Qwen-1.8B 23.11.30 32K ✅ 2.2T 5.8GB 2.9GB ✅
Qwen-7B 23.08.03 32K ❎ 2.4T 11.5GB 8.2GB ✅
Qwen-14B 23.09.25 8K ❎ 3.0T 18.7GB 13.0GB ✅
Qwen-72B 23.11.30 32K ✅ 3.0T 61.4GB 48.9GB ✅

In this repo, you can figure out:

Also, if you meet problems, turn to FAQ for help first. Still feeling struggled? Feel free to shoot us issues (better in English so that more people can understand you)! If you would like to help us, send us pull requests with no hesitation! We are always excited about PR!

Would like to chat with us or date us coffee time? Welcome to our Discord or WeChat!

News and Updates


Performance

Qwen models outperform the baseline models of similar model sizes on a series of benchmark datasets, e.g., MMLU, C-Eval, GSM8K, MATH, HumanEval, MBPP, BBH, etc., which evaluate the models’ capabilities on natural language understanding, mathematic problem solving, coding, etc. Qwen-72B achieves better performance than LLaMA2-70B on all tasks and outperforms GPT-3.5 on 7 out of 10 tasks.


Model MMLU C-Eval GSM8K MATH HumanEval MBPP BBH CMMLU
5-shot 5-shot 8-shot 4-shot 0-shot 3-shot 3-shot 5-shot
LLaMA2-7B 46.8 32.5 16.7 3.3 12.8 20.8 38.2 31.8
LLaMA2-13B 55.0 41.4 29.6 5.0 18.9 30.3 45.6 38.4
LLaMA2-34B 62.6 - 42.2 6.2 22.6 33.0 44.1 -
ChatGLM2-6B 47.9 51.7 32.4 6.5 - - 33.7 -
InternLM-7B 51.0 53.4 31.2 6.3 10.4 14.0 37.0 51.8
InternLM-20B 62.1 58.8 52.6 7.9 25.6 35.6 52.5 59.0
Baichuan2-7B 54.7 56.3 24.6 5.6 18.3 24.2 41.6 57.1
Baichuan2-13B 59.5 59.0 52.8 10.1 17.1 30.2 49.0 62.0
Yi-34B 76.3 81.8 67.9 15.9 26.2 38.2 66.4 82.6
XVERSE-65B 70.8 68.6 60.3 - 26.3 - - -
Qwen-1.8B 45.3 56.1 32.3 2.3 15.2 14.2 22.3 52.1
Qwen-7B 58.2 63.5 51.7 11.6 29.9 31.6 45.0 62.2
Qwen-14B 66.3 72.1 61.3 24.8 32.3 40.8 53.4 71.0
Qwen-72B 77.4 83.3 78.9 35.2 35.4 52.2 67.7 83.6

For all compared models, we report the best scores between their official reported results and OpenCompass.

For more experimental results (detailed model performance on more benchmark datasets) and details, please refer to our technical report by clicking here.

Requirements


Quickstart

Below, we provide simple examples to show how to use Qwen-Chat with 🤖 ModelScope and 🤗 Transformers.

You can use our pre-built docker images to skip most of the environment setup steps, see Section "Using Pre-built Docker Images" for more details.

If not using docker, please make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.

pip install -r requirements.txt

If your device supports fp16 or bf16, we recommend installing flash-attention (we support flash attention 2 now.) for higher efficiency and lower memory usage. (flash-attention is optional and the project can run normally without installing it)

git clone https://github.com/Dao-AILab/flash-attention
cd flash-attention && pip install .
# Below are optional. Installing them might be slow.
# pip install csrc/layer_norm
# If the version of flash-attn is higher than 2.1.1, the following is not needed.
# pip install csrc/rotary

Now you can start with ModelScope or Transformers.

🤗 Transformers

To use Qwen-Chat for the inference, all you need to do is to input a few lines of codes as demonstrated below. Remember to pass in the correct model names or paths, such as "Qwen/Qwen-7B-Chat" and "Qwen/Qwen-14B-Chat". However, please make sure that you are using the latest code.

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

# Model names: "Qwen/Qwen-7B-Chat", "Qwen/Qwen-14B-Chat"
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)

# use bf16
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, bf16=True).eval()
# use fp16
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
# use cpu only
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="cpu", trust_remote_code=True).eval()
# use auto mode, automatically select precision based on the device.
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen-7B-Chat",
    device_map="auto",
    trust_remote_code=True
).eval()

# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.
# model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)

# 1st dialogue turn
response, history = model.chat(tokenizer, "你好", history=None)
print(response)
# 你好！很高兴为你提供帮助。

# 2nd dialogue turn
response, history = model.chat(tokenizer, "给我讲一个年轻人奋斗创业最终取得成功的故事。", history=history)
print(response)
# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。
# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。
# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。
# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。
# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。
# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。

# 3rd dialogue turn
response, history = model.chat(tokenizer, "给这个故事起一个标题", history=history)
print(response)
# 《奋斗创业：一个年轻人的成功之路》

Running Qwen, the base language model, is also simple.

Running Qwen 
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

# Model names: "Qwen/Qwen-7B", "Qwen/Qwen-14B" 
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B", trust_remote_code=True)
# use bf16
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B", device_map="auto", trust_remote_code=True, bf16=True).eval()
# use fp16
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B", device_map="auto", trust_remote_code=True, fp16=True).eval()
# use cpu only
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B", device_map="cpu", trust_remote_code=True).eval()
# use auto mode, automatically select precision based on the device.
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen-7B",
    device_map="auto",
    trust_remote_code=True
).eval()

# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.
# model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B", trust_remote_code=True)

inputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是', return_tensors='pt')
inputs = inputs.to(model.device)
pred = model.generate(**inputs)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...

In the event of a network issue while attempting to download model checkpoints and codes from HuggingFace, an alternative approach is to initially fetch the checkpoint from ModelScope and then load it from the local directory as outlined below: 

from modelscope import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

# Downloading model checkpoint to a local dir model_dir
# model_dir = snapshot_download('qwen/Qwen-7B')
# model_dir = snapshot_download('qwen/Qwen-7B-Chat')
# model_dir = snapshot_download('qwen/Qwen-14B')
model_dir = snapshot_download('qwen/Qwen-14B-Chat')

# Loading local checkpoints
# trust_remote_code is still set as True since we still load codes from local dir instead of transformers
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    device_map="auto",
    trust_remote_code=True
).eval()

🤖 ModelScope

ModelScope is an open-source platform for Model-as-a-Service (MaaS), which provides flexible and cost-effective model service to AI developers. Similarly, you can run the models with ModelScope as shown below:

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

# Model names: "qwen/Qwen-7B-Chat", "qwen/Qwen-14B-Chat"
tokenizer = AutoTokenizer.from_pretrained("qwen/Qwen-7B-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参

response, history = model.chat(tokenizer, "你好", history=None)
print(response)
response, history = model.chat(tokenizer, "浙江的省会在哪里？", history=history) 
print(response)
response, history = model.chat(tokenizer, "它有什么好玩的景点", history=history)
print(response)

Batch Inference

Qwen supports batch inference. With flash attention enabled, using batch inference can bring a 40% speedup. The example code is shown below:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import GenerationConfig
from qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids

# To generate attention masks automatically, it is necessary to assign distinct
# token_ids to pad_token and eos_token, and set pad_token_id in the generation_config.
tokenizer = AutoTokenizer.from_pretrained(

Qwen (QwenLM/Qwen)

Searching Database…

21.9K ·GitHub
Qwen
这是什么？Qwen(通义千问)的官方仓库——阿里云推出的对话及预训练大语言模型系列，在 Hugging Face 和 ModelScope 上发布了从 1.8B 到 72B 参数的多个规模版本。
这些是下载下来自己运行的模型，还是一个在线聊天机器人服务？是模型权重而不是服务：你可以从 Hugging Face 或 ModelScope 下载 Qwen/Qwen-7B-Chat 这类检查点，自己用 Transformers 来运行——或者通过预构建的 Docker 镜像来运行。
我怎么才能把这些模型跑起来？用几行 Transformers 或 ModelScope 代码就能加载检查点；预构建的 Docker 镜像可以省去大部分环境配置。flash-attention 在 fp16/bf16 设备上是一个可选的提速项。
在运行之前需要装好哪些东西？需要 Python 3.8+、PyTorch 1.12+(推荐 2.0+)、transformers 4.32+,使用 GPU 的用户还需要 CUDA 11.4+。flash-attention 是可选的——没有它项目也能正常运行。
这个仓库还是获取 Qwen 的正确地方吗？这个仓库已经冻结：由于代码库差异很大，团队已迁移到 QwenLM/Qwen2,不再维护它。它仍是最初 Qwen-1.8B/7B/14B/72B 版本的所在地。
它到底能在哪里运行——只能跑在 Linux 服务器上，还是技术栈支持的地方都行？只要 PyTorch 技术栈能跑的地方都可以，并且提供了预构建的 Docker 镜像；除 NVIDIA GPU 之外，还支持在昇腾 910 NPU 和海光 DCU 加速卡上做推理。
模型下载好之后，所有东西都在我自己的硬件上运行吗？是的——推理完全在本地进行。如果从 Hugging Face 下载不稳定，你可以改从 ModelScope 获取检查点，然后从本地目录加载。
我的显卡比较一般——模型有哪些规模，有没有更轻量的版本？规模从 1.8B 到 72B 不等。对话模型还提供 Int4 和 Int8 量化版本——Qwen-7B-Chat-Int4 显存占用少得多、运行也更快，而基准测试成绩没有明显下降。
Qwen-72B 和 LLaMA2-70B 或 GPT-3.5 相比怎么样？根据 README 中的基准测试表格，Qwen-72B 在所有列出的任务上都超过了 LLaMA2-70B(MMLU 77.4,C-Eval 83.3,HumanEval 35.4),并且在 10 项任务中有 7 项优于 GPT-3.5。
我可以用自己的数据对它进行微调吗？可以——README 中记录了针对 Qwen-7B 模型的全参数微调，以及 LoRA 和 Q-LoRA 方法。
这些模型的上下文窗口有多长？Qwen-72B 和 Qwen-72B-Chat 支持 32k 上下文，更新后的 Qwen-7B 上下文长度也从 2048 扩展到了 8192 个 token。
它的英文能力是一流的吗，还是主要面向中文的模型？两种语言都是一流的：基准测试同时覆盖英文(72B 的 MMLU 为 77.4)和中文(C-Eval 83.3,CMMLU 83.6),而且持续更新也在不断提升较小模型的中文知识水平。
What are similar projects?Qwen3-ASR, airllm, petals, jeeves, ChatGLM2-6B, jeff, Hy-MT2, openvino
Which topics is it in?finetuning, flash-attention, long-context, alibaba, open-weights, pretrained-models, natural-language-processing, transformers, qwen, quantization, large-language-models, multilingual
Creator
QwenLM
Size
36.5 MB
Stars
21.9K
Updated
CommitMar 05
Age
3y
Commits · 90 Days