midscene README

The project's README from github.com/web-infra-dev/midscene. As one markdown file: README.md.

Midscene.js
GUI Agent for E2E Testing
AI-powered vision. Cross-platform. Batteries included.

Website · English / 简体中文

npm version downloads License Featured on GitHub Trending

See it in action

Midscene combines a vision-driven GUI Agent with a testing kit for writing, verifying, and debugging UI tests across web, mobile, and desktop apps through the same Agent APIs. The Playwright example below shows a web test; configure a model and open your app in an existing Playwright page to get started:

import { PlaywrightAgent } from '@midscene/web/playwright';

const agent = new PlaywrightAgent(page);

// Let the Agent carry out a flow, then verify the result.
await agent.aiAct('Search for headphones, then filter the results to under $100');
await agent.aiWaitFor('The filtered search results are displayed');
await agent.aiAssert('Every product in the search results has a price below $100');

Open the generated HTML report to inspect screenshots, actions, and assertion results. Follow the Playwright guide for model setup, a complete example, and test runner integration.

👁️ GUI Agent

Midscene models both UI actions and assertions on how people use software: look at the screen, act on what you see, and check the visible result. Describe the task and expected outcome in natural language; Midscene uses screenshots to decide where to interact and whether the interface meets your expectations.

Visual understanding and cross-platform actions

Like a person finding a control on screen, Midscene locates elements by their appearance and position, then clicks, types, or scrolls to carry out your instructions. You can target icon-only buttons, custom controls, <canvas>, and elements inside cross-origin iframes without writing selectors or adding semantic annotations.

The same Agent APIs work across Web, Android, iOS, HarmonyOS, and desktop apps. You can also connect a custom interface by providing screenshot and action capabilities.

Verify what users see

Assertions follow the same visual approach: Midscene inspects the screen as a human tester would to judge whether the expected result is visible. Describe the expected appearance in natural language to check colors, selection highlights, layout, and visual feedback — including content drawn on <canvas> or displayed in native apps.

await agent.aiAssert('The selected plan has a blue border and a checkmark');
await agent.aiAssert('The error message is visible below the email field');

Benchmark performance

Benchmark Pass@1 Model used in the reported run
AndroidWorld 93.1% Gemini-3.5-Flash
MobileWorld 78.6% Gemini-3.6-Flash
AppControlBench 96.7% Doubao Seed 2.1 Turbo

Each report includes the run configuration and task results; see AndroidWorld's report for its environment and validator adjustments.

Cost and model choice

Screenshot-based UI actions avoid sending large DOM trees to the model. In the reported AppControlBench run, Midscene with Doubao Seed 2.1 Turbo completed the 60-task evaluation with $0.59 in total model cost, achieving 58 passes. The report provides per-task costs and comparisons across models.

Midscene supports multimodal models such as Qwen3.x, Doubao-Seed-2.1, GLM-4.6V, gemini-3.5-flash, and UI-TARS, including open-source options you can self-host. Start with one model, or combine planning and vision models for your workload. For data extraction and page understanding, you can opt in to include DOM. See Model Strategy.

Showcases

🧰 Testing Kit

Batteries included: Midscene provides the test framework, observability, and integration APIs needed to turn GUI automation into a maintainable E2E test project.

Midscene Test: an E2E framework for the AI era

Midscene Test (@midscene/test, Beta) separates declarative test intent from programmable engineering. Write UI flows and expectations in YAML, and wrap API calls, data preparation, and cleanup in reusable TypeScript Nodes. A refund test can prepare an order through an API, request a refund through the UI, and verify the result in one workflow.

The framework includes project scaffolding, platform presets, lifecycle hooks, retries, and execution-project isolation and concurrency. It also generates a Markdown reference from registered Nodes and their parameter schemas, so people and AI Agents can discover the same capabilities and co-maintain test cases. See Create and extend a project and Write and run tests.

Built-in observability

Interactive HTML reports show screenshots, element locations, the AI decision process, and action and assertion results. Midscene Test records the inputs, outputs, duration, and status of each AI step and custom business operation. Reports and runtime logs give both developers and AI Agents the context to investigate failures. Use the Playground to try and refine instructions against your interface.

Rich APIs that fit your existing stack

Use aiAct for autonomous flows, aiTap and aiInput for individual actions, aiAssert for assertions, and aiQuery for structured data extraction. Combine these Agent APIs with your existing code, fixtures, and assertions through Playwright, Puppeteer, or the JavaScript SDK. You can adopt Midscene's visual capabilities within your current testing framework. AI coding agents can also operate interfaces through Midscene Skills.

🚀 Get started

📄 Resources

🤝 Community

🌟 Awesome Midscene

Community projects that extend Midscene.js capabilities:

📝 Credits

We would like to thank the following projects:

📖 Citation

If you use Midscene.js in your research or project, please cite:

@software{Midscene.js,
  author = {Xiao Zhou, Tao Yu, YiBing Lin},
  title = {Midscene.js: GUI Agent for E2E Testing.},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/web-infra-dev/midscene}
}

✨ Star History

Star History Chart

📝 License

Midscene.js is MIT licensed.

If this project helps you or inspires you, please give us a star

midscene (web-infra-dev/midscene)

Searching Database…

15.3K ·GitHub
Midscene
这是什么？Midscene 是一个基于视觉的 GUI 智能体，结合了 E2E 测试工具包：它通过查看截图来操作和验证界面，借助同一套 Agent API 覆盖 Web、移动端和桌面应用。
这是 Playwright 插件，还是独立的智能体？它是一个独立的 Agent 框架（aiAct、aiTap、aiInput、aiAssert、aiQuery），可以接入 Playwright、Puppeteer 或 JS SDK，也可以通过 Midscene Test 用 YAML 以声明式方式驱动，它并不是普通的 Playwright 插件。
我该如何开始编写测试？可以先在 Playground 或 Chrome 扩展程序中试用，然后用 Playwright/Puppeteer SDK 编写测试，或者用 Midscene Test 以 YAML 编写测试，并配置模型，将其指向已有的 Playwright 页面。
我需要自己准备 AI 模型吗？需要。你要配置一个多模态模型，例如 Qwen3.x、Doubao-Seed-2.1、GLM-4.6V、gemini-3.5-flash 或 UI-TARS；开源选项也可以自行部署，无需使用付费 API。
它是否像 Selenium 选择器那样依赖页面 DOM？不依赖。操作和断言主要基于截图，以避免把庞大的 DOM 树发送给模型；如果是提取数据的任务，你可以选择加入 DOM 数据。
它除了能测试网页，也能测试移动端和桌面应用吗？可以。同一套 Agent API 适用于 Web、Android、iOS、HarmonyOS 和桌面应用；社区 SDK 还扩展了 iOS Mirror、Windows/macOS/Linux PC 控制（包括 Docker 镜像）、Python 和 Java 支持。
Midscene 提供哪些核心 API？aiAct 驱动自主的多步骤流程，aiTap 和 aiInput 执行单个操作，aiAssert 检查视觉预期，aiQuery 从屏幕上提取结构化数据。
它在真实基准测试中表现如何？据公布的 Pass@1 分数，AndroidWorld 上为 93.1%（Gemini-3.5-Flash），MobileWorld 上为 78.6%（Gemini-3.6-Flash），AppControlBench 上为 96.7%（Doubao Seed 2.1 Turbo）；完整运行 60 个任务的 AppControlBench 测试，模型费用约为 $0.59。
AI 编程智能体也能用 Midscene 操作界面吗？可以。Midscene Skills 让 AI 编程智能体使用相同的视觉 Agent 能力来操作界面，而不只是运行人工编写的测试脚本。
什么是 Midscene Test？它是一个测试版的 E2E 框架，把用 YAML 编写的界面流程和预期结果，与可复用的 TypeScript Node 分开。Node 用于 API 调用、数据准备和清理；框架同时提供脚手架、重试机制，以及可用 Node 能力的自动生成文档。
What are similar projects?mobile-mcp, testsprite-cli, UI-TARS-desktop, e2e, typesafe-computer-use, agent-desktop, collab-agent, playwright
Which topics is it in?gui-agent, computer-use-agent, mobile-automation, web-automation, qa, browser-use, desktop-automation, e2e-testing, gui-automation, test-automation, computer-use, yaml
Creator
web-infra-dev
Size
764.3 MB
Stars
15.3K
Updated
CommitOct 16Releasev1.16.0Oct 16
Age
2y
Commits · 90 Days