UI-TARS README

The project's README from github.com/bytedance/UI-TARS. As one markdown file: README.md.

Local Image

🌐 Website   | 🤗 Hugging Face Models   |    🔧 Deployment    |    📑 Paper    |   🖥️ UI-TARS-desktop  
🏄 Midscene (Browser Automation)    |   🫨 Discord  

We also offer a UI-TARS-desktop version, which can operate on your local personal device. To use it, please visit https://github.com/bytedance/UI-TARS-desktop. To use UI-TARS in web automation, you may refer to the open-source project Midscene.js. ❗Notes: Since Qwen 2.5vl based models ultilizes absolute coordinates to ground objects, please kindly refer to our illustration about how to process coordinates in this guide.

Updates

Introduction

UI-TARS-1.5, an open-source multimodal agent built upon a powerful vision-language model. It is capable of effectively performing diverse tasks within virtual worlds.

Leveraging the foundational architecture introduced in our recent paper, UI-TARS-1.5 integrates advanced reasoning enabled by reinforcement learning. This allows the model to reason through its thoughts before taking action, significantly enhancing its performance and adaptability, particularly in inference-time scaling. Our new 1.5 version achieves state-of-the-art results across a variety of standard benchmarks, demonstrating strong reasoning capabilities and notable improvements over prior models.

🚀 Quick Start Guide: Deploying and Using Our Model

To help you get started quickly with our model, we recommend following the steps below in order. These steps will guide you through deployment, prediction post-processing to make the model take actions in your environment.

✅ Step 1: Deployment & Inference

👉 Deployment and Inference. This includes instructions for model deployment using huggingface endpoint, and running your first prediction.

✅ Step 2: Post Processing

Installation

pip install ui-tars
# or
uv pip install ui-tars

Usage

from ui_tars.action_parser import parse_action_to_structure_output, parsing_response_to_pyautogui_code

response = "Thought: Click the button\nAction: click(start_box='(100,200)')"
original_image_width, original_image_height = 1920, 1080
parsed_dict = parse_action_to_structure_output(
    response,
    factor=1000,
    origin_resized_height=original_image_height,
    origin_resized_width=original_image_width,
    model_type="qwen25vl"
)
print(parsed_dict)
parsed_pyautogui_code = parsing_response_to_pyautogui_code(
    responses=parsed_dict,
    image_height=original_image_height,
    image_width=original_image_width
)
print(parsed_pyautogui_code)
FYI: Coordinates visualization

To help you better understand the coordinate processing, we also provide a guide for coordinates processing visualization.

Prompt Usage Guide

To accommodate different device environments and task complexities, the following three prompt templates in codes/ui_tars/prompt.py. are designed to guide GUI agents in generating appropriate actions. Choose the template that best fits your use case:

🖥️ COMPUTER_USE

Recommended for: GUI tasks on desktop environments such as Windows, Linux, or macOS.

Features:

📱 MOBILE_USE

Recommended for: GUI tasks on mobile devices or Android emulators.

Features:

📌 GROUNDING

Recommended for: Lightweight tasks focused solely on action output, or for use in model training and evaluation.

Features:

When developing or evaluating multimodal interaction systems, choose the appropriate prompt template based on your target platform (desktop vs. mobile)

Performance

Online Benchmark Evaluation

Benchmark type Benchmark UI-TARS-1.5 OpenAI CUA Claude 3.7 Previous SOTA
Computer Use OSworld (100 steps) 42.5 36.4 28 38.1 (200 step)
Windows Agent Arena (50 steps) 42.1 - - 29.8
Browser Use WebVoyager 84.8 87 84.1 87
Online-Mind2web 75.8 71 62.9 71
Phone Use Android World 64.2 - - 59.5

Grounding Capability Evaluation

Benchmark UI-TARS-1.5 OpenAI CUA Claude 3.7 Previous SOTA
ScreenSpot-V2 94.2 87.9 87.6 91.6
ScreenSpotPro 61.6 23.4 27.7 43.6

Poki Game

Model 2048 cubinko energy free-the-key Gem-11 hex-frvr Infinity-Loop Maze:Path-of-Light shapes snake-solver wood-blocks-3d yarn-untangle laser-maze-puzzle tiles-master
OpenAI CUA 31.04 0.00 32.80 0.00 46.27 92.25 23.08 35.00 52.18 42.86 2.02 44.56 80.00 78.27
Claude 3.7 43.05 0.00 41.60 0.00 0.00 30.76 2.31 82.00 6.26 42.86 0.00 13.77 28.00 52.18
UI-TARS-1.5 100.00 0.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00

Minecraft

Task Type Task Name VPT DreamerV3 Previous SOTA UI-TARS-1.5 w/o Thought UI-TARS-1.5 w/ Thought
Mine Blocks (oak_log) 0.8 1.0 1.0 1.0 1.0
(obsidian) 0.0 0.0 0.0 0.2 0.3
(white_bed) 0.0 0.0 0.1 0.4 0.6
200 Tasks Avg. 0.06 0.03 0.32 0.35 0.42
Kill Mobs (mooshroom) 0.0 0.0 0.1 0.3 0.4
(zombie) 0.4 0.1 0.6 0.7 0.9
(chicken) 0.1 0.0 0.4 0.5 0.6
100 Tasks Avg. 0.04 0.03 0.18 0.25 0.31

Model Scale Comparison

Here we compare performance across different model scales of UI-TARS on the OSworld benchmark.

Benchmark Type Benchmark UI-TARS-72B-DPO UI-TARS-1.5-7B UI-TARS-1.5
Computer Use OSWorld 24.6 27.5 42.5
GUI Grounding ScreenSpotPro 38.1 49.6 61.6

Limitations

While UI-TARS-1.5 represents a significant advancement in multimodal agent capabilities, we acknowledge several important limitations:

What's next

We are providing early research access to our top-performing UI-TARS-1.5 model to facilitate collaborative research. Interested researchers can contact us at [email protected].

Looking ahead, we envision UI-TARS evolving into increasingly sophisticated agentic experiences capable of performing real-world actions, thereby empowering platforms such as doubao to accomplish more complex tasks for you :)

Star History

Star History Chart

Citation

If you find our paper and model useful in your research, feel free to give us a cite.

@article{qin2025ui,
  title={UI-TARS: Pioneering Automated GUI Interaction with Native Agents},
  author={Qin, Yujia and Ye, Yining and Fang, Junjie and Wang, Haoming and Liang, Shihao and Tian, Shizuo and Zhang, Junda and Li, Jiahao and Li, Yunxin and Huang, Shijue and others},
  journal={arXiv preprint arXiv:2501.12326},
  year={2025}
}

UI-TARS (bytedance/UI-TARS)

Searching Database…

11.6K ·GitHub
UI-TARS
What is this?UI-TARS is an open-source multimodal agent model built on a vision-language model that can operate GUIs directly — clicking, typing, scrolling, dragging — across desktop apps, mobile screens, browsers, and even video games like Minecraft, reasoning about its actions before taking them.
Is this a ready-to-use app or a model I have to deploy myself?It is a vision-language model plus deployment and post-processing code that you host yourself (e.g. via a HuggingFace inference endpoint); there is also a separate 'UI-TARS-desktop' companion project for running it as an app on your own device.
How do I get it running?Follow the two-step Quick Start Guide: Step 1 deploy the model and run your first prediction via a HuggingFace inference endpoint, then Step 2 install and run the post-processing code (codes/uitars/prompt.py) that turns model predictions into real GUI actions.
Do I need an Anthropic or OpenAI API key, or is the model self-contained?No external API key is needed — UI-TARS is its own open-sourced model (UI-TARS-1.5-7B) that you deploy yourself; benchmarks in the README simply compare it against OpenAI CUA and Claude 3.7.
What are the known limitations?The authors are explicit: it can be misused to bypass protections like CAPTCHA, needs substantial compute for large or long tasks, can hallucinate or misidentify GUI elements, and the open-sourced 7B model isn't specifically optimized for game-playing scenarios.
Does it only work on desktop, or mobile too?It supports both: the COMPUTERUSE prompt template handles desktop actions on Windows, Linux, or macOS, while MOBILEUSE adds Android-style actions like longpress, openapp, and pressback for phone and emulator use.
Does running this require serious hardware?Yes, the README states it still needs substantial computational resources, especially for large-scale tasks or extended gameplay sessions.
What is UI-TARS-desktop?A separate companion project that lets you run UI-TARS as a ready-made agent directly on your own local computer instead of deploying the model and post-processing code yourself.
Can it actually play video games?Yes — it's benchmarked on 14 Poki browser games and on Minecraft tasks like mining blocks and killing mobs, hitting 100% on most Poki games where OpenAI CUA and Claude 3.7 score far lower.
How does it stack up against OpenAI's and Claude's computer-use agents?On the OSWorld benchmark UI-TARS-1.5 scores 42.5 versus 36.4 for OpenAI CUA and 28 for Claude 3.7, and on the ScreenSpotPro grounding benchmark it scores 61.6 versus 23.4 and 27.7 respectively.
What is doubao, and why is it mentioned?Doubao is ByteDance's AI platform; the team envisions UI-TARS eventually powering it to carry out more complex real-world tasks for users.
What are similar projects?OmniParser, Agent-S, ZDTaichu5.0-9B, Nex-N2.5, fara, Valen, InternVL, midscene
Which topics is it in?gui-agent, minecraft, vlm, computer-use, autonomous-agents, reinforcement-learning, browser-automation, multimodal, gaming, huggingface, computer-vision, benchmark
Creator
bytedance
Size
43.3 MB
Stars
11.6K
Updated
CommitSep 05
Age
1y
Commits · 90 Days