Qwen-Drive-1.0 README

The project's README from github.com/QwenLM/Qwen-Drive-1.0. As one markdown file: README.md.

Qwen-Drive

An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

Qwen Team  ·  Huazhong University of Science and Technology

📑 Technical Report  |  📖 Blog  |  🤗 Hugging Face  |  🤖 ModelScope

Qwen-Drive-1.0 performance overview

Welcome to the GitHub repository of Qwen-Drive-1.0. Here you can find official information about Qwen-Drive, and post your questions (Issues).

Introduction

Qwen-Drive-1.0 retains the architecture of the pretrained Qwen3.5 vision-language model and integrates 3D perception, visual question answering, and motion planning within a unified framework. The natively multimodal Qwen3.5-4B serves as the shared VLM, with two external modules attached:

We propose a staged training strategy that integrates perception, language, and planning objectives. By combining driving-specific supervision with general-purpose vision-language data, the model achieves specialized driving competence while retaining broad visual understanding and instruction-following capabilities. This approach is supported by a unified data pipeline that: (1) maps heterogeneous perception annotations into a shared label space; (2) re-annotates driving VQA responses to ensure format and factual consistency; and (3) standardizes trajectories from multiple public driving datasets into a unified waypoint representation.

Qwen-Drive-1.0 unified architecture

3D perception results Planning results

Performance

Planning

SFT RL
NAVSIM v1.1 navtest, PDMS 88.2 (89.3 best-of-6) 90.7 (91.4 best-of-6)
Waymo Open Dataset E2E test, RFS 7.78 7.91
NVIDIA PhysicalAI open-loop, minADE 3 s 0.34 m 0.38 m

SFT is the imitation-trained Planning Expert. RL is the same expert after reward optimization on the benchmark objectives. Both share one VLM. 3D detection, occupancy and map segmentation results are in the technical report, full planning tables in docs/evaluation.md.

Driving VQA

LingoQA Ego3D RMSE ↓ VLAD SURDS WaymoQA safety WaymoQA all CoC all IH
InternVL3.5-8B-Instruct 46.4 23.01 54.5 32.8 54.5 58.1 – 47.5
LLaVA-OV2-8B 41.2 24.97 58.7 38.6 49.7 55.2 0.6 54.0
Qwen3.5-4B 70.4 13.17 65.4 53.0 62.5 67.1 2.6 59.0
Cosmos-Reason1-7B 45.2 26.71 33.6 8.5 39.5 43.9 3.2 30.5
Cosmos-Reason2-8B 59.6 12.62 56.4 19.5 57.7 57.9 1.7 56.0
Cosmos3-nano 65.0 22.41 57.7 39.7 56.9 58.4 4.0 2.0
MiMo-Embodied-7B 72.0 9.85 50.3 43.1 66.5 69.6 – 61.0
Alpamayo-1.5-10B 64.0 25.31 9.1 3.1 42.6 44.4 3.4 3.0
Qwen-Drive-1.0-SFT 77.8 7.78 66.5 66.1 70.7 74.5 41.3 71.0

LingoQA is scored with Qwen-Plus as the judge instead of the official LingoJudge, which we found to score leniently and inconsistently across scenarios. Under the official LingoJudge protocol Qwen-Drive-1.0-SFT obtains a LingoScore of 79.4. – marks an invalid or unparsable response.

On driving-scene understanding, Qwen-Drive-1.0 improves markedly over its Qwen3.5-4B base while keeping general vision-language ability intact. Driving QA and spatial understanding, with overall causal-reasoning accuracy on PAI-AV Chain-of-Causation (CoC) and an in-house Chinese urban driving-decision set (IH):

General VQA and Reasoning:

MMBench MMStar MMMU MMMU-Pro std MMMU-Pro vis CharXiv OCRBench RealWorldQA SimpleVQA CountQA
InternVL3.5-8B-Instruct 80.0 64.1 62.0 46.4 42.3 41.7 83.2 66.9 40.8 20.9
LLaVA-OV2-8B 82.7 64.9 54.7 36.3 26.0 40.1 79.3 71.8 36.7 22.6
Qwen3.5-4B 87.1 75.3 73.4 64.9 61.3 65.1 86.9 76.3 47.8 35.9
Cosmos-Reason1-7B 80.0 63.5 54.2 38.4 35.8 39.7 85.2 67.5 45.0 18.5
Cosmos-Reason2-8B 82.8 65.3 59.1 36.1 43.5 42.5 87.0 67.5 45.3 22.3
Cosmos3-nano 79.6 66.7 60.9 46.4 40.8 42.1 85.2 69.7 45.0 23.6
MiMo-Embodied-7B – 22.4 – 27.4 28.1 57.5 78.8 28.5 – 22.6
Alpamayo-1.5-10B 7.5 26.1 27.4 15.6 13.5 1.5 3.2 46.9 – 4.7
Qwen-Drive-1.0-SFT 85.5 75.9 72.7 62.7 59.7 64.4 86.4 79.0 46.1 31.7

Spatial Understanding and Grounding:

EmbSpatial ERQA RefSpatial Omni3D ODinW13
InternVL3.5-8B-Instruct 74.2 42.0 – – –
LLaVA-OV2-8B 78.4 42.3 – – –
Qwen3.5-4B 76.0 46.3 54.5 47.4 40.8
Cosmos-Reason1-7B 68.8 38.5 0.4 – 4.8
Cosmos-Reason2-8B 77.6 43.3 51.8 32.9 40.2
Cosmos3-nano 77.9 41.3 – 32.3 35.9
MiMo-Embodied-7B 45.1 39.8 2.2 – –
Alpamayo-1.5-10B 20.6 27.5 – – –
Qwen-Drive-1.0-SFT 78.9 48.5 50.8 45.8 45.9

Comparisons are reproduced under one protocol with near-deterministic decoding; see the technical report for the full setup.

Models

The model can be downloaded from Hugging Face or ModelScope. Everything ships in one directory. The VLM sits at its root, shared by every task, and each task head in a subfolder beside it.

Qwen-Drive-1.0-4B/          9.1 GB  the VLM, which on its own serves the VQA mode
├── planner-sft/            2.1 GB  Planning Expert, imitation-trained
├── planner-rl/             2.1 GB  Planning Expert after reward optimization
└── perception/             0.5 GB  BEV perception head

A head is attached when the VLM is loaded:

model = QwenDriveForPlanning.from_pretrained(
    "Qwen-Drive-1.0-4B", planner="Qwen-Drive-1.0-4B/planner-rl", dtype=torch.bfloat16
)

planner-rl was reward-optimized only on reasoning-conditioned rollouts, so run it in the reasoning planning mode. planner-sft covers both direct and reasoning planning.

Install

A GPU with 24 GB+ of memory is recommended.

git clone <repository-url> qwen-drive && cd qwen-drive

# Any Python virtual environment works; conda is shown here
conda create -n qwen-drive python=3.10
conda activate qwen-drive

pip install -e. --no-build-isolation        # or: pip install -r requirements.txt

Quick start

data/demo/ bundles four WOD-E2E planning scenes with their frames in one Parquet file and six perception frames, so the commands below need nothing but the weights. The scenes cover a night intersection whose light turns green, a left turn, a right turn, and a slow-down past a parked truck.

scripts/demo.py --plot writes a summary figure: the camera ring of the scene on the left, one row per view and one column per timestep, the predicted trajectories against the ground truth on the right, and the generated reasoning underneath.

export PYTHONPATH=src
python scripts/demo.py --model Qwen-Drive-1.0-4B --planner Qwen-Drive-1.0-4B/planner-rl \
    --scenes data/demo/planning_scenes.jsonl --image-archive data/demo/frames.parquet \
    --plot demo.png

import torch
from qwen_drive import InferenceMode, QwenDriveForPlanning
from qwen_drive.benchmarks import read_scene_file
from qwen_drive.images import ImageArchive

model = QwenDriveForPlanning.from_pretrained(
    "Qwen-Drive-1.0-4B",
    planner="Qwen-Drive-1.0-4B/planner-rl",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).to("cuda").eval()

scene = next(
    read_scene_file(
        "data/demo/planning_scenes.jsonl",
        image_archive=ImageArchive.open("data/demo/frames.parquet"),
    )
).scene

result = model.run(InferenceMode.REASONING_PLANNING, scene=scene, num_samples=6)
print(result.reasoning)
print(result.trajectories.shape)   # (6, 50, 3) -> (x, y, heading), 5 s at 10 Hz

docs/cookbook.md has recipes for VQA, best-of-N planning, benchmark runs, multi-GPU evaluation, perception inference and visualization.

Documentation

Doc Contents
docs/cookbook.md recipes for every inference mode and benchmark
docs/model.md architecture, configuration fields, decoding parameters
docs/data.md scene-file format, obtaining the frames, frame packing
docs/evaluation.md benchmark protocols, metric definitions, result tables
docs/perception.md perception setup, demo data layout, coordinate conventions

Repository layout

qwen-drive/
├── src/qwen_drive/             # VQA + planning: model, scenes, benchmarks, metrics
├── src/qwen_drive_perception/  # perception mode (+ CUDA kernels under ops/)
├── scripts/                    # demo, prediction and scoring, visualization
├── data/demo/                  # bundled demo scenes and perception frames
├── data/benchmarks/            # the four benchmark scene files (relative frame paths)
├── assets/                     # figures
└── docs/

Citation

If you find our work helpful, feel free to cite us.

@misc{zhou2026qwendrive10initialstepvisionlanguage,
      title={Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving}, 
      author={Xin Zhou and Zongchuang Zhao and Zhibo Yang and Mingsheng Li and Humen Zhong and Shuai Bai and Du Chu and Ruizhe Chen and Zhaohai Li and Jun Tang and Qiuyue Wang and Mingkun Yang and Jiazhao Zhang and Dayiheng Liu and Dingkang Liang and Xiang Bai},
      year={2026},
      eprint={2609.00111},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.00111}, 
}

License

Qwen-Drive-1.0 is released under the Apache 2.0 license. Please find more details in the LICENSE file.

Qwen-Drive-1.0 (QwenLM/Qwen-Drive-1.0)

vlm
Searching Database…

515 ·GitHub
Qwen-Drive-1.0
What is this?Qwen-Drive-1.0 is a vision-language foundation model for autonomous driving, built on the Qwen3.5-4B VLM and extended with 3D perception, visual question answering, and motion planning in one unified model.
Is this a ready-to-drive self-driving stack or a research model I run myself?It is a released model checkpoint plus inference code: a shared VLM with a BEV Perception Head for 3D detection/occupancy/map segmentation and a Planning Expert for trajectory generation, meant for benchmarking and research rather than deployment in a car.
How do I try it out?Download the weights from Hugging Face or ModelScope, then run 'scripts/demo.py --plot' on the bundled demo scenes to get a figure with camera views, predicted trajectories vs ground truth, and generated reasoning.
Do I need a beefy GPU to run this?Yes, the authors recommend a GPU with at least 24 GB of memory to run the model.
Is there one planning mode or do I need to pick the right one?There are two planner variants: planner-sft covers both direct and reasoning-based planning, while planner-rl was reward-optimized only for reasoning-conditioned rollouts and must be run in reasoning planning mode.
Who is this built for?It targets autonomous-driving and vision-language researchers who want a unified model that handles 3D perception, driving-scene VQA, and trajectory planning for benchmarking against other driving VLMs.
How does Qwen-Drive-1.0 compare to other driving VLMs like InternVL3.5 or Cosmos-Reason?It outperforms baselines such as InternVL3.5-8B, LLaVA-OV2-8B, and the Cosmos-Reason models on driving VQA benchmarks (e.g. LingoQA 77.8 vs 46.4-72.0) and on NAVSIM/Waymo planning metrics, while keeping general VQA performance close to its Qwen3.5-4B base.
What's the model architecture under the hood?A single Qwen3.5-4B VLM backbone is shared across tasks: a BEV Perception Head handles 3D detection, occupancy, and map segmentation, a Planning Expert predicts future ego trajectories, and the unchanged LLM decoder handles general and driving VQA.
What demo data comes with the repo?data/demo/ includes four Waymo Open Dataset E2E planning scenes (a night intersection, a left turn, a right turn, a slow-down past a parked truck) plus six perception frames, all in one Parquet file, so you can run the demo without collecting your own data.
Is there a paper behind this model?Yes, the README links to a technical report with full architecture, training, and evaluation details, and points to docs/evaluation.md for complete planning result tables.
What are similar projects?ZDTaichu5.0-9B, damo-radar, OmniParser, carla, sam3, VisionHOPE, OraRL, EvoVLM
Which topics is it in?self-driving-car, autonomous-driving, vision-language-models, autonomous-vehicles, vision, segmentation, vlm, foundation-models, vision-language-model, object-detection, qwen, multimodal
Creator
QwenLM
Size
59.2 MB
Stars
515
Updated
CommitSep 03
Age
1mo
Commits · 90 Days