What is this?
Agent S is an open-source agent framework that controls a real desktop GUI - clicking, typing, and scrolling - to complete natural-language tasks the way a human would, without per-app API integrations.
Is this a ready-made app or something I wire into my own code?
It's a Python framework/library (the gui-agents package) with both a CLI and an SDK (the AgentS3 class) - you run it from the terminal or import it into your own scripts. Sai is a separate, hosted commercial product built on the same research if you'd rather not run anything yourself.
How do I install and run it?
Install the gui-agents package with pip, or clone the repo and install it locally if you want to modify the code; on macOS also run brew install tesseract for OCR support. Then launch the CLI with flags like --provider, --model, --groundprovider, --groundurl and --groundmodel, or drive the AgentS3 class directly from the SDK.
Which models do I need to supply?
A main reasoning model (OpenAI, Anthropic, Gemini, Azure OpenAI, Open Router, or vLLM) plus a separate grounding model that turns screen understanding into click coordinates - the README recommends UI-TARS-1.5-7B hosted on Hugging Face Inference Endpoints.
What are the big warnings before I let it loose on my machine?
The agent runs real Python code to control your computer, and its optional local coding environment executes arbitrary Python/Bash with your own user permissions - the README says to only use it in trusted environments. It's also designed for single-monitor setups only.
Is this meant for personal automation or is it a research project?
It's primarily a research framework for building and benchmarking computer-use agents on tests like OSWorld; for a polished, managed agent the authors point users to their commercial product Sai instead.
Does it work the same on Mac, Windows, and Linux?
Yes, Agent S3 officially supports Linux, Mac, and Windows.
What is OSWorld, and how good is Agent S3 on it?
OSWorld is a benchmark of long, realistic desktop tasks; Agent S3 alone scores 66% and climbs to 72.6% with Behavior Best-of-N, surpassing the roughly 72% human baseline.
What is Sai, and how does it relate to this repo?
Sai is Simular's separate hosted, production computer-use agent built on Agent S's ideas; it reached 73% on the newer OSWorld 2.0 benchmark, ahead of GPT-5.6 Sol, and targets people who want a managed service instead of running the research framework themselves.
Why does it need two different models?
The main model does high-level planning from screenshots and instructions, while a separate grounding model (like UI-TARS-1.5-7B) converts that plan into precise pixel coordinates for clicks and keystrokes.
Can it run code instead of just clicking around?
Yes - an optional local coding environment lets the agent call a callcodeagent action to execute Python or Bash directly for tasks like file manipulation or data processing, though it runs with the same permissions as your user account.
What are similar projects?
self-operating-computer, fara, trae-agent, Agentlas-OS, qwen-code, turnstone, Raven, hive
Which topics is it in?
grounding, cua, planning, computer-use-agent, azure, retrieval-augmented-generation, openrouter, computer-use, ocr, vllm, autonomous-agents, memory
Agent S:
Open Computer Use