Agent-S README

The project's README from github.com/simular-ai/Agent-S. As one markdown file: README.md.

Logo Agent S: Use Computer Like a Human

An open-source computer use agent that operates a real GUI — mouse, keyboard, and screen — the way a person does.

🏆 Sai: Tops OSWorld 2.0, Beating GPT and Opus with Lower Costs (73%)

🏆 Agent S3: First to Surpass Human Performance on OSWorld (72.60%)

  🌐 [Sai OSWorld 2.0 report]  🌐 [S3 blog]  📄 [S3 Paper (TMLR 2026)]  🎥 [S3 Video]

  🌐 [S2 blog]  📄 [S2 Paper (COLM 2025)]  🎥 [S2 Video]

  🌐 [S1 blog]  📄 [S1 Paper (ICLR 2025)]  🎥 [S1 Video]

  simular-ai%2FAgent-S | Trendshift

Windows macOS Linux Discord    PyPI Downloads

Deutsch | Español | français | 日本語 | 한국어 | Português | Русский | 中文
  

Skip the setup? Try Agent S in Simular Cloud

🥳 Updates

Table of Contents

  1. 💡 Introduction
  2. 🎯 OSWorld Benchmark Results
  3. 🛠️ Installation & Setup
  4. 🚀 Usage
  5. 🎬 See Sai in Action
  6. 🤝 Acknowledgements
  7. 💬 Citation

💡 Introduction

Agent S is an open source computer use agent framework from Simular. It takes a natural-language task, looks at the screen, and completes the task by clicking, typing, and scrolling in ordinary desktop and web applications — no API integration and no per-app scripting required. Agent S3 is the third generation of the framework and was the first computer use agent to surpass human performance on the OSWorld benchmark, at 72.60%. It runs on macOS, Windows, and Linux, and works with models from OpenAI, Anthropic, and open-weight providers.

The research continues in production: in August 2026, Sai — Simular's hosted computer use agent, built on the ideas in this repo — reached a 73% success rate on OSWorld 2.0, ahead of GPT-5.6 Sol at 62.57% as reported by OpenAI, and at lower cost. If you are looking for a hosted, production version of this technology rather than a research framework, see Sai.

Whether you're researching OS agents, automating your own desktop, or contributing to open source computer use, we're excited to have you here.

🎯 OSWorld Benchmark Results

August 2026 — Sai tops OSWorld 2.0 (73%)

Our flagship product Sai reaches a 73% success rate on OSWorld 2.0, a 108-task benchmark of long professional and everyday tasks that take skilled humans more than an hour to complete. That places Sai ahead of GPT-5.6 Sol at 62.57% as reported by OpenAI, and it does so at lower cost per task. Full write-up: Sai Tops OSWorld 2.0.

Agent S3 on OSWorld, WindowsAgentArena and AndroidWorld

Agent S3 Results

On OSWorld, Agent S3 alone reaches 66% in the 100-step setting, already exceeding the previous state of the art of 63.4% (GTA1 w/ GPT-5). With the addition of Behavior Best-of-N, performance climbs even higher to 72.6%, surpassing human-level performance on OSWorld (~72%)!

Agent S3 also demonstrates strong zero-shot generalization! On WindowsAgentArena, accuracy rises from 50.2% using only Agent S3 to 56.6% by selecting from 3 rollouts. Similarly on AndroidWorld, performance improves from 68.1% to 71.6%

🛠️ Installation & Setup

Prerequisites

Installation

To install Agent S3 without cloning the repository, run

pip install gui-agents

If you would like to test Agent S3 while making changes, clone the repository and install using

pip install -e .

Don't forget to also brew install tesseract! Pytesseract requires this extra installation to work.

API Configuration

Option 1: Environment Variables

Add to your .bashrc (Linux) or .zshrc (MacOS):

export OPENAI_API_KEY=<YOUR_API_KEY>
export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY>
export HF_TOKEN=<YOUR_HF_TOKEN>

Option 2: Python Script

import os
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

Supported Models

We support Azure OpenAI, Anthropic, Gemini, Open Router, and vLLM inference. See models.md for details.

Grounding Models (Required)

For optimal performance, we recommend UI-TARS-1.5-7B hosted on Hugging Face Inference Endpoints or another provider. See Hugging Face Inference Endpoints for setup instructions.

🚀 Usage

⚡️ Recommended Setup:
For the best configuration, we recommend using OpenAI gpt-5-2025-08-07 as the main model, paired with UI-TARS-1.5-7B for grounding.

CLI

Note, this is running Agent S3, our improved agent, without bBoN.

Run Agent S3 with the required parameters:

agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080

Local Coding Environment (Optional)

For tasks that require code execution (e.g., data processing, file manipulation, system automation), you can enable the local coding environment:

agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080 \
    --enable_local_env

⚠️ WARNING: The local coding environment executes arbitrary Python and Bash code locally on your machine. Only use this feature in trusted environments and with trusted inputs.

Required Parameters

Optional Parameters

Grounding Model Dimensions

The grounding width and height should match the output coordinate resolution of your grounding model:

Optional Parameters

Local Coding Environment Details

The local coding environment enables Agent S3 to execute Python and Bash code directly on your machine. This is particularly useful for:

When enabled, the agent can use the call_code_agent action to execute code blocks for tasks that can be completed through programming rather than GUI interaction.

Requirements:

Security Considerations:

gui_agents SDK

First, we import the necessary modules. AgentS3 is the main agent class for Agent S3. OSWorldACI is our grounding agent that translates agent actions into executable python code.

import pyautogui
import io
from gui_agents.s3.agents.agent_s import AgentS3
from gui_agents.s3.agents.grounding import OSWorldACI
from gui_agents.s3.utils.local_env import LocalEnv  # Optional: for local coding environment

# Load in your API keys.
from dotenv import load_dotenv
load_dotenv()

current_platform = "linux"  # "darwin", "windows"

Next, we define our engine parameters. engine_params is used for the main agent, and engine_params_for_grounding is for grounding. For engine_params_for_grounding, we support custom endpoints like HuggingFace TGI, vLLM, and Open Router.

engine_params = {
  "engine_type": provider,
  "model": model,
  "base_url": model_url,           # Optional
  "api_key": model_api_key,        # Optional
  "temperature": model_temperature # Optional
}

# Load the grounding engine from a custom endpoint
ground_provider = "<your_ground_provider>"
ground_url = "<your_ground_url>"
ground_model = "<your_ground_model>"
ground_api_key = "<your_ground_api_key>"

# Set grounding dimensions based on your model's output coordinate resolution
# UI-TARS-1.5-7B: grounding_width=1920, grounding_height=1080
# UI-TARS-72B: grounding_width=1000, grounding_height=1000
grounding_width = 1920  # Width of output coordinate resolution
grounding_height = 1080  # Height of output coordinate resolution

engine_params_for_grounding = {
  "engine_type": ground_provider,
  "model": ground_model,
  "base_url": ground_url,
  "api_key": ground_api_key,  # Optional
  "grounding_width": grounding_width,
  "grounding_height": grounding_height,
}

Then, we define our grounding agent and Agent S3.

# Optional: Enable local coding environment
enable_local_env = False  # Set to True to enable local code execution
local_env = LocalEnv() if enable_local_env else None

grounding_agent = OSWorldACI(
    env=local_env,  # Pass local_env for code execution capability
    platform=current_platform,
    engine_params_for_generation=engine_params,
    engine_params_for_grounding=engine_params_for_grounding,
    width=1920,  # Optional: screen width
    height=1080  # Optional: screen height
)

agent = AgentS3(
    engine_params,
    grounding_agent,
    platform=current_platform,
    max_trajectory_length=8,  # Optional: maximum image turns to keep
    enable_reflection=True     # Optional: enable reflection agent
)

Finally, let's query the agent!

# Get screenshot.
screenshot = pyautogui.screenshot()
buffered = io.BytesIO() 
screenshot.save(buffered, format="PNG")
screenshot_bytes = buffered.getvalue()

obs = {
  "screenshot": screenshot_bytes,
}

instruction = "Close VS Code"
info, action = agent.predict(instruction=instruction, observation=obs)

exec(action[0])

Refer to gui_agents/s3/cli_app.py for more details on how the inference loop works.

OSWorld

To deploy Agent S3 in OSWorld, follow the OSWorld Deployment instructions.

🎬 See Sai in Action

Most computer work is not one-off — it repeats. Sai turns those repeating jobs into recurring workflows: describe the routine once, set when it should run, and the agent does it on a cloud computer on your schedule. Browse ready-made routines in the Sai workflow template gallery, or start from the Sai API if you would rather trigger runs from your own code.

💬 Citations

If you find this codebase useful, please cite:

@misc{Agent-S3,
      title={The Unreasonable Effectiveness of Scaling Agents for Computer Use}, 
      author={Gonzalo Gonzalez-Pumariega and Vincent Tu and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2510.02250},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2510.02250}, 
}

@misc{Agent-S2,
      title={Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents}, 
      author={Saaket Agashe and Kyle Wong and Vincent Tu and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2504.00906},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.00906}, 
}

@inproceedings{Agent-S,
    title={{Agent S: An Open Agentic Framework that Uses Computers Like a Human}},
    author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang},
    booktitle={International Conference on Learning Representations (ICLR)},
    year={2025},
    url={https://arxiv.org/abs/2410.08164}
}

Star History

Star History Chart

Agent-S (simular-ai/Agent-S)

Searching Database…

12.6K ·GitHub
Agent-S
这是什么？Agent S 是一个开源智能体框架，它可以控制真实的桌面图形界面，包括点击、输入和滚动，从而像人一样完成自然语言描述的任务，无需为每个应用单独对接 API。
这是现成的应用，还是需要我自己接入代码的工具？它是一个 Python 框架/库（gui-agents 包），同时提供命令行工具（CLI）和 SDK（AgentS3 类）。你可以在终端中运行，也可以把它导入自己的脚本。如果你不想自己运行任何东西，Sai 是基于同一研究成果开发的独立托管商业产品。
该如何安装和运行？用 pip 安装 gui-agents 包；如果想修改代码，可以克隆仓库并在本地安装。在 macOS 上，还需运行 brew install tesseract 以支持 OCR 功能。随后可使用 --provider、--model、--groundprovider、--groundurl 和 --groundmodel 等参数启动命令行工具，或直接通过 SDK 调用 AgentS3 类。
我需要自己提供哪些模型？需要一个主推理模型（可选 OpenAI、Anthropic、Gemini、Azure OpenAI、Open Router 或 vLLM），以及一个独立的定位（grounding）模型，用于将屏幕理解转换为点击坐标。README 推荐在 Hugging Face Inference Endpoints 上托管 UI-TARS-1.5-7B。
在让它在我的电脑上运行之前，有哪些重要警告？该智能体会运行真实的 Python 代码来控制你的电脑。它可选的本地编程环境会以你当前用户的权限执行任意 Python/Bash 代码。README specifies 只应在受信任的环境中使用。此外，它仅支持单显示器配置。
这是用于个人自动化的，还是一个研究项目？它主要是一个研究框架，用于构建和评测计算机操作智能体，测试基准包括 OSWorld 等。如果想要成熟的托管式智能体，作者建议改用他们的商业产品 Sai。
在 Mac、Windows 和 Linux 上的使用体验是否一致？是的，Agent S3 官方支持 Linux、Mac 和 Windows。
OSWorld 是什么？Agent S3 在上面表现如何？OSWorld 是一个包含长时、真实桌面任务的基准测试。Agent S3 单独得分 66%，使用 Behavior Best-of-N 后提升至 72.6%，超过了约 72% 的人类基线。
Sai 是什么？它与本仓库有什么关系？Sai 是 Simular 独立推出的托管式生产级计算机操作智能体，基于 Agent S 的思路开发。它在较新的 OSWorld 2.0 基准上达到 73%，超过了 GPT-5.6 Sol。它面向希望使用托管服务、而不是自行运行研究框架的用户。
为什么需要两个不同的模型？主模型负责根据截图和指令进行高层规划，而独立的定位模型（如 UI-TARS-1.5-7B）则将规划转换为精确的像素坐标，用于点击和键盘输入。
它能运行代码，而不只是点来点去吗？可以。它可选的本地编程环境允许智能体调用 callcodeagent 操作，直接执行 Python 或 Bash，用于文件处理或数据处理等任务，但它与你的用户账户拥有相同的权限。
What are similar projects?self-operating-computer, fara, trae-agent, bytebot, qwen-code, Raven, turnstone, hive
Which topics is it in?cua, planning, computer-use-agent, retrieval-augmented-generation, azure, computer-use, openrouter, vllm, ocr, autonomous-agents, memory, multimodal
Creator
simular-ai
Size
59.1 MB
Stars
12.6K
Updated
CommitOct 08Releasev0.3.2Dec 16
Age
2y
Commits · 90 Days