Topics / #gui-automation

GUI automation projects on GitHub

Automating interactions with graphical user interfaces, such as clicking, typing and screen inspection.

As of , the most-starred GUI automation projects on GitHub are agent-zero (19.5K stars), midscene (15.3K stars) and E2B (14.4K stars). GitHub.Locker tracks 24 GUI automation repositories with 161.5K stars combined. The fastest-growing on October 18 was iPhone-use, with +157 stars in a day.

24Projects
161.5KStars in total
+363Stars in a day
Browse in the catalog →github.com/topics/gui-automation ↗

Hot right now

Most stars gained in a day · Oct 18

+157

iPhone-use

zhongerxin/iPhone-use · ★ 4.2K

iPhone Use lets Codex control your real iPhone over USB using natural language: it can open apps, read screens, tap, scroll, type text, and organize lists, while showing a live screen preview in the sidebar.

+42

mobile-mcp

mobile-next/mobile-mcp · ★ 9.8K

Mobile MCP is a Model Context Protocol server that lets AI agents automate and scrape native iOS and Android apps - on simulators, emulators, or real devices - through a single platform-agnostic set of tools.

+41

phone-harness

ShawnPana/phone-harness · ★ 3.8K

Phone Harness lets AI coding agents like Claude Code or Codex control a real phone — an iPhone through the Mac's iPhone Mirroring window, an Android over adb, or a rented cloud phone — with no jailbreak and nothing installed on the phone itself.

+32

Operit

AAswordman/Operit · ★ 8.8K

Operit is an open-source AI agent platform for Android that connects cloud or local AI models to your phone's system, terminal, browser, files, and project workspaces so the AI can actually perform tasks like file management, coding, and device automation.

The all-time top

24 projects · by stars

  1. #1 · agent0ai/agent-zero

    agent-zero

    Agent Zero is an open-source agent framework that gives your AI agent a full Dockerized Linux computer — a real desktop, browser, documents and plugins — instead of just a chat window.

    19.5K
  2. #2 · web-infra-dev/midscene

    midscene

    Midscene is a vision-driven GUI Agent combined with an E2E testing kit: it looks at screenshots to act on and verify UI across web, mobile, and desktop apps through one set of Agent APIs.

    15.3K
  3. #3 · e2b-dev/E2B

    E2B

    E2B is open-source infrastructure that lets you run AI-generated code in secure, isolated cloud sandboxes, so agents can execute untrusted code safely.

    14.4K
  1. 4

    Auto.js

    12.9Kclearw5/Auto.js

    Auto.js is an Android-based JavaScript coding platform built for learning to program and automating tasks directly on an Android device.

  2. 5

    robotjs

    12.8Koctalmage/robotjs · C

    RobotJS is a Node.js library for desktop automation: it controls the mouse and keyboard, and reads pixels from the screen.

  3. 6

    pyautogui

    12.7Kasweigart/pyautogui · Python

    PyAutoGUI is a cross-platform Python module that lets you programmatically control the mouse and keyboard to automate GUI interactions.

  4. 7

    robotgo

    10.9Kgo-vgo/robotgo · Go

    A Go library for native cross-platform desktop automation: it drives the mouse and keyboard, reads the screen, manages windows and processes, and listens to global events — aimed at RPA, auto-testing and AI computer use.

  5. 8

    self-operating-computer

    10.3KOthersideAI/self-operating-computer · Python

    A framework that lets a multimodal AI model operate your computer: it views the screen and issues mouse and keyboard actions to reach the objective you give it. One of the first full computer-use projects (Nov 2023).

  6. 9

    easytrader

    10.3Kshidenggui/easytrader · Python

    easytrader is a Python library for automating Chinese stock quant trading - it can drive the 同花顺 (THS) desktop client, use the official miniqmt quant API, rebalance or track 雪球 (Xueqiu) portfolios, and mirror joinquant/ricequant paper trading.

  7. 10

    mobile-mcp

    9.8Kmobile-next/mobile-mcp · TypeScript

    Mobile MCP is a Model Context Protocol server that lets AI agents automate and scrape native iOS and Android apps - on simulators, emulators, or real devices - through a single platform-agnostic set of tools.

  8. 11

    Operit

    8.8KAAswordman/Operit · Kotlin

    Operit is an open-source AI agent platform for Android that connects cloud or local AI models to your phone's system, terminal, browser, files, and project workspaces so the AI can actually perform tasks like file management, coding, and device automation.

  9. 12

    astron-rpa

    5.3Kiflytek/astron-rpa · Python

    AstronRPA is an open-source, enterprise-grade Robotic Process Automation (RPA) desktop application with a visual designer that lets you build low-code/no-code workflows to automate desktop software and web pages, with native integration into iFlytek's Astron Agent platform.

  10. 13

    iPhone-use

    4.2Kzhongerxin/iPhone-use · Python

    iPhone Use lets Codex control your real iPhone over USB using natural language: it can open apps, read screens, tap, scroll, type text, and organize lists, while showing a live screen preview in the sidebar.

  11. 14

    MaaEnd

    4.1KMaaEnd/MaaEnd · C++

    MaaEnd is a vision-AI-based automation assistant for the game 'Arknights: Endfield' (明日方舟:终末地), built on MaaFramework, MXU and MaaFwApp.

  12. 15

    phone-harness

    3.8KShawnPana/phone-harness · Python

    Phone Harness lets AI coding agents like Claude Code or Codex control a real phone — an iPhone through the Mac's iPhone Mirroring window, an Android over adb, or a rented cloud phone — with no jailbreak and nothing installed on the phone itself.

  13. 16

    agent-desktop

    1.8Klahfir/agent-desktop · Rust

    Agent Desktop is a Rust-built tool that gives AI agents reliable control over desktop apps by reading the OS accessibility tree instead of screenshots, so actions target real UI elements with stable, retry-safe references instead of guessed pixel coordinates.

  14. 17

    typesafe-computer-use

    1.2Kawlevin/typesafe-computer-use · Python

    A Python agent that drives your Mac toward a goal you type in plain English: it OCRs the screen, a cheap classifier picks each action, and it clicks — about $0.0002 per step.

  15. 18

    rewards-farmer

    1KUser0332/rewards-farmer · Python

    rewards-farmer automates Microsoft Rewards: it drives Microsoft Edge through a webdriver to run Bing searches and complete the daily visual-search task so you rack up reward points without doing it by hand.

  16. 19

    dsh-vision-toolkit

    899Anionex/dsh-vision-toolkit · TypeScript

    DSH Vision Toolkit is a plugin for DeepSeek Harness (DSH) that gives text-only AI models 'eyes': paste or reference an image and the agent can answer questions about it, OCR long screenshots, rebuild a UI from a mockup, locate elements, crop, vectorize, or diff pixels.

  17. 20

    deepseek-harness-android-app

    513woaiys3/deepseek-harness-android-app · Java

    It is an installable Android APK that packages DeepSeek Harness (DSH), an AI agent core, so the AI can directly operate your phone — install apps, tap the screen, change settings, read/edit files — without needing root (via Shizuku) or with root if available.

  18. 21

    dingusppc

    410dingusdev/dingusppc · C++

    DingusPPC is an experimental emulator for PowerPC Macintosh computers, aiming to accurately emulate Old World and early New World PowerPC Macs (and the Pippin) with a focus on accuracy and debugging rather than just compatibility.

  19. 22

    huashu-mac-use

    311alchaincyf/huashu-mac-use · Swift

    An Agent Skill that lets coding agents like Claude Code, Codex and Cursor drive macOS apps that have no API — reading screens, clicking and typing — and leaves reproducible evidence of every step.

  20. 23

    solari-cookbook

    191solari-sdk/solari-cookbook · Python

    A cookbook of small, runnable example programs for Solari, a service that gives you cloud browsers, sandboxes, and desktop VMs behind a single API key.

  21. 24

    Codex-Minecraft-Gameplay

    182wz1119/Codex-Minecraft-Gameplay · Python

    A Windows toolkit that lets Codex or other computer-use agents play Minecraft by reading the screen and sending keyboard and mouse input to move, gather, craft, explore, and build.

Often tagged together