WeVisOne-VL README

项目的 README,来自 github.com/Tencent/WeVisOne-VL。合并为一个 markdown 文件:README.md。

WeChat Vision

WeVisOne-VL

Unified Visual Perception from Objects to Pixels

Project Page Paper WeVisOne-VL-2B WeVisOne-VL-4B 2B Demo 4B Demo License

English | 简体中文

WeVisOne-VL (WeChat Vision All-in-One) is a vision-language foundation model that unifies object-level perception (object detection, referring localization, counting, pointing, and OCR) and pixel-level perception (referring segmentation and promptable alpha matting) through structured next-token prediction and a lightweight 9.6M-parameter pixel decoder—enabling localization at any user-specified granularity from points and boxes to binary masks and continuous alpha mattes, while reliably rejecting absent-target queries.

Overview of WeVisOne-VL

✨ Highlights

📢 News

📝 TODO

🏆 Performance at a Glance

Headline numbers for WeVisOne-VL-4B against the strongest open baseline in each benchmark (Qwen3-VL / Qwen3.5 up to 9B, Rex-Omni, LocateAnything, and task-specific segmentation / matting models).

Task Benchmark Metric Best Baseline WeVisOne-VL-4B
📦 Detection LVIS F1 Mean ↑ 50.7 55.3
📦 Dense Detection Dense200 F1 Mean ↑ 58.7 71.3
💬 Referring RefCOCOg test F1 Mean ↑ 77.6 81.8
🚫 Rejection WeRef-Bench Rej. Rate ↑ 73.0 96.4
🔢 Counting PixMo-Count test Acc ↑ 82.8 89.6
📍 Pointing PointBench Acc ↑ 69.9 71.9
🔤 OCR TotalText F1 Mean ↑ 43.3 52.0
🎭 Referring Seg. ReasonSeg val gIoU ↑ 65.0 71.2
✂️ Box Matting P3M-500-NP MAD ↓ 15.4 7.3
✂️ Text Matting WeRefMat (Referring) MAD ↓ 22.4 9.8

📊 Full Benchmark Results (click each task to expand)

F1 for detection, referring and OCR is averaged over IoU thresholds 0.5:0.95 ("Mean"). Greedy decoding, ≤ 1,960 visual tokens per image.

Object Detection (F1 Mean)
Method COCO LVIS Dense200 VisDrone
GroundingDINO-T 56.6 38.8 33.1 38.5
Qwen3-VL-8B 47.8 42.9 13.8 29.5
Qwen3.5-9B 49.5 43.7 18.0 31.3
Rex-Omni-3B 52.9 46.9 58.3 35.8
LocateAnything-3B 54.7 50.7 58.7 39.9
WeVisOne-VL-2B 55.2 53.3 61.4 40.4
WeVisOne-VL-4B 57.1 55.3 71.3 44.7
Referring Object Detection (F1 Mean / Rejection Rate %)
Method RefCOCOg test RefCOCOg Rej WeRef Mean WeRef Rej HumanRef Ref-L4
Qwen3-VL-8B 74.9 22.5 36.7 18.3 71.6 72.4
Qwen3.5-9B 74.6 15.2 50.7 70.7 77.8 72.0
Rex-Omni-3B 74.3 0.5 50.3 43.3 79.9 65.2
LocateAnything-3B 77.6 0.6 51.7 73.0 78.7 74.3
WeVisOne-VL-2B 78.2 50.2 73.2 92.9 77.4 75.1
WeVisOne-VL-4B 81.8 67.4 78.7 96.4 80.7 78.1
Object Counting (Exact-match Accuracy %)
Method CountQA PixMo-Count test CLOC test KubriCount testB MixCount test
Qwen3.5-9B 41.6 80.4 22.4 19.1 7.0
Rex-Omni-3B 37.6 71.9 22.3 18.8 11.4
LocateAnything-3B 52.4 82.8 23.9 22.4 15.2
WeVisOne-VL-2B 47.9 88.1 26.9 32.4 32.2
WeVisOne-VL-4B 57.1 89.6 28.3 40.1 40.4
Object Pointing
Method PointBench PixMo-Points COCO LVIS Dense200 VisDrone
Qwen3-VL-8B 62.5 72.2 77.5 67.7 42.2 46.4
Rex-Omni-3B 69.9 72.9 80.5 70.8 82.5 58.9
LocateAnything-3B 56.9 68.6 83.9 76.6 87.6 60.4
WeVisOne-VL-2B 62.9 73.2 85.8 78.9 83.2 62.4
WeVisOne-VL-4B 71.9 74.7 85.4 79.5 87.0 64.4
OCR — Text Grounding & Recognition (F1 Mean)
Method HierText ICDAR2015 TotalText SROIE
PP-OCRv5 (specialist) 30.5 25.6 25.7 58.6
Qwen3-VL-8B 25.7 27.1 35.4 43.3
Rex-Omni-3B 28.0 28.1 40.6 44.8
LocateAnything-3B 29.4 24.1 43.3 40.3
WeVisOne-VL-2B 30.5 42.6 42.5 62.1
WeVisOne-VL-4B 39.0 47.8 52.0 68.1
Referring Segmentation (cIoU; ReasonSeg uses gIoU)
Method RefCOCO testA RefCOCO+ testA RefCOCOg test ReasonSeg val Ref-ZOM cIoU Ref-ZOM Rej
LISA-13B (ft) 78.8 70.2 70.5 65.0 - -
GSVA-7B 78.9 69.6 73.3 50.5 66.6 41.9
UniPixel-7B 83.0 80.1 77.1 60.5 62.9 0.3
Youtu-VL-4B 82.0 79.6 76.6 - 61.0 1.3
WeVisOne-VL-2B 82.9 79.0 74.9 60.9 73.4 62.4
WeVisOne-VL-4B 84.5 81.1 77.8 71.2 72.2 86.0
Promptable Matting (MAD ×10³ ↓)
Method P3M-500-NP (Box) AM-2K (Box) PPM-100 (Box) WeRefMat (Category) WeRefMat (Referring)
Matte Anything 24.4 13.2 10.5 - -
SAM2-Matting (SAM3) 15.4 12.1 10.8 70.8 51.9
Qwen3.5-9B + SAM2-Matting - - - 77.8 22.4
WeVisOne-VL-2B 6.6 5.7 5.3 17.7 15.8
WeVisOne-VL-4B 7.3 5.0 6.1 16.4 9.8

See the technical report for all splits and metrics, and evaluation/README.md to reproduce the public benchmarks.

🏗️ How It Works

1. Data Engine. A knowledge-graph-driven engine discovers concepts, localizes them with complementary experts verified by a VLM, and extends the verified boxes to masks and alpha mattes. Over 100M images are annotated this way.

Data engine

2. Unified Task Formulation. Every task is written as structured text, with coordinate tokens <0> … <1000> for boxes and points, None for absent targets, and an <OBJ> token after each box for pixel-level outputs.

Unified task formulation

3. Model Architecture. A 9.6M-parameter pixel decoder combines the hidden state of each <OBJ> token, its predicted box and multi-level Qwen3-VL ViT features to predict a full-resolution mask or alpha matte.

Pixel decoder architecture

🚀 Quick Start

Installation

git clone https://github.com/Tencent/WeVisOne-VL.git && cd WeVisOne-VL
conda create -n wevisone-vl python=3.10 -y && conda activate wevisone-vl
pip install -r requirements.txt && pip install -v -e .

Inference

from PIL import Image
from wevisone_vl import WeVisOneVL
from wevisone_vl.visualization import save_result

model = WeVisOneVL("tencent/WeVisOne-VL-4B", backend="transformers")  # or backend="vllm"
# model = WeVisOneVL("tencent/WeVisOne-VL-2B", backend="transformers")  # Use 2B instead

# Object-level: open-vocabulary detection
image = Image.open("demo/examples/images/livingroom.jpg").convert("RGB")
result = model.predict(image, task="detection", query=["sofa", "pillow", "coffee table", "lamp"])
save_result("outputs/detection", image, result)

# Pixel-level: promptable alpha matting
image = Image.open("demo/examples/images/seabreeze.jpg").convert("RGB")
result = model.predict(image, task="matting", query="person")
save_result("outputs/matting", image, result)

Input · query="person"

Predicted alpha matte

Cutout

Supported Tasks

task query Output
📦 detection category list boxes
💬 referring referring expression boxes or None
📍 point category / expression points
🔢 count_box / count_point category / expression boxes / points + count
🔤 ocr / ocr_grounding – / text to find text + boxes
🎭 segmentation category / expression boxes + binary masks
✂️ matting category / expression boxes + alpha mattes

[!TIP] device ("auto", "cuda:0", "npu:0", "cpu"), dtype, min_pixels / max_pixels, and the usual decoding arguments (max_new_tokens, temperature, ...) are also available. The Cookbooks walk through each task in detail.

📓 Cookbooks

Tutorial Covers
📦 Object-level pipeline Detection, Referring, Counting, Pointing
🔤 OCR pipeline OCR, Text Grounding
🎭 Pixel-level pipeline Segmentation, Matting

🎮 Gradio Demo

Try the hosted demo: 2B / 4B

Launch the demo locally:

CUDA_VISIBLE_DEVICES=0 python demo/app.py --model tencent/WeVisOne-VL-4B
# Or use 2B:
# CUDA_VISIBLE_DEVICES=0 python demo/app.py --model tencent/WeVisOne-VL-2B
More launch options
# vLLM backend on a custom port, with a public share link
CUDA_VISIBLE_DEVICES=0 python demo/app.py \
    --model tencent/WeVisOne-VL-4B --backend vllm \
    --server-name 0.0.0.0 --server-port 8080 --share

📊 Evaluation

See evaluation/README.md.

🖼️ Representative Visualizations

Side-by-side comparisons with strong baselines. Click a task to expand or collapse it.

Referring Detection — locating the objects described by a natural-language expression
Referring detection comparison
Box-prompted Matting — predicting the alpha matte of the target inside a given box
Box-prompted matting comparison
Text-prompted Matting — predicting the alpha matte of the target specified by a category name or referring expression
Text-prompted matting comparison
Object Detection — locating all instances of the given categories with boxes
Object detection comparison
Object Counting — counting the instances that match a category or expression
Object counting comparison
Object Pointing — marking each queried object with a point
Object pointing comparison
OCR — locating and recognizing text in the image
OCR comparison
Referring Segmentation — segmenting the objects described by a natural-language expression
Referring segmentation comparison

📜 License

Tencent-authored code in this repository is released under the Apache License 2.0. Third-party components keep their original licenses; please check the corresponding source files.

📖 Citation

If WeVisOne-VL is helpful to your work, please consider giving us a ⭐ and citing:

@article{liang2026wevisone,
  title={WeVisOne-VL: Unified Visual Perception from Objects to Pixels},
  author={Liang, Tianming and Pan, Junyi and Li, Yuxi and Fu, Canmiao and Huang, Zhipeng and Sun, Chong and Li, Chen and Lyu, Jing},
  journal={arXiv preprint arXiv:2610.31329},
  year={2026}
}

WeVisOne-VL (Tencent/WeVisOne-VL)

正在搜索数据库…

这是什么?WeVisOne-VL 是腾讯的 WeChat Vision Team 推出的视觉语言模型。同一个模型既能处理对象级任务(检测、指代、计数、指向、OCR),也能处理像素级任务(分割、alpha 抠图)。
这是我要下载的模型,还是需要调用的服务?需要自己下载权重:2B 和 4B 模型在 Hugging Face 上,用推理包来运行。README 提到了托管的 Gradio 演示,但没有提供托管 API。
怎么把它跑起来?可以先试用托管的 Gradio 演示(2B 或 4B),无需安装任何东西。若要在自己的代码里使用,请安装推理包,按照 Quick Start 操作,再参考 Cookbooks,其中详细介绍了每项任务。
运行前需要准备什么?需要 Hugging Face 上的模型权重(2B 或 4B)和推理包。Cookbooks 是 Jupyter notebook,所以需要 Jupyter 才能跟着操作。README 没有给出版本锁定信息。
现在能微调吗?目前仓库里还做不到。微调代码仍在 TODO 清单中未勾选,所以只能按原样使用已发布的 2B 和 4B 权重。
能在 CPU 上运行吗,还是必须用 CUDA?device 选项接受 'auto'、'cuda:0'、'npu:0' 或 'cpu',因此支持 CPU 推理。README 没有给出显存数据,所以在选择硬件前请先查看模型卡(model card)。
它会凭空生成图中不存在的物体吗?设计上不会。对于不存在的目标,它返回 None,而不是编造结果;4B 模型在 WeRef-Bench 上的拒绝率为 96.4。
获取掩码和抠图还需要 SAM 吗?不需要。一个 9.6M 参数的像素解码器可以直接输出全分辨率的掩码和 alpha 抠图,因此无需 SAM。
4B 模型在密集检测上表现如何?在 Dense200 检测上,4B 模型达到 F1 71.3,而 README 主要结果表中最佳基线为 58.7。
Mask2Alpha 是什么?一个抠图标注工具,已在 mask2alpha 分支开源,可把二值掩码转换为高质量的 alpha 抠图。它曾被用于 WeVisOne-VL 的 Data Engine。
作者
Tencent
大小
30.4 MB
星标
1.2K
更新
提交Oct 25
时长
27天
提交 · 90 天