midscene README

The project's README from github.com/web-infra-dev/midscene. As one markdown file: README.md.

Midscene.js
GUI Agent for E2E Testing
AI-powered vision. Cross-platform. Batteries included.

Website · English / 简体中文

npm version downloads License Featured on GitHub Trending

See it in action

Midscene combines a vision-driven GUI Agent with a testing kit for writing, verifying, and debugging UI tests across web, mobile, and desktop apps through the same Agent APIs. The Playwright example below shows a web test; configure a model and open your app in an existing Playwright page to get started:

import { PlaywrightAgent } from '@midscene/web/playwright';

const agent = new PlaywrightAgent(page);

// Let the Agent carry out a flow, then verify the result.
await agent.aiAct('Search for headphones, then filter the results to under $100');
await agent.aiWaitFor('The filtered search results are displayed');
await agent.aiAssert('Every product in the search results has a price below $100');

Open the generated HTML report to inspect screenshots, actions, and assertion results. Follow the Playwright guide for model setup, a complete example, and test runner integration.

👁️ GUI Agent

Midscene models both UI actions and assertions on how people use software: look at the screen, act on what you see, and check the visible result. Describe the task and expected outcome in natural language; Midscene uses screenshots to decide where to interact and whether the interface meets your expectations.

Visual understanding and cross-platform actions

Like a person finding a control on screen, Midscene locates elements by their appearance and position, then clicks, types, or scrolls to carry out your instructions. You can target icon-only buttons, custom controls, <canvas>, and elements inside cross-origin iframes without writing selectors or adding semantic annotations.

The same Agent APIs work across Web, Android, iOS, HarmonyOS, and desktop apps. You can also connect a custom interface by providing screenshot and action capabilities.

Verify what users see

Assertions follow the same visual approach: Midscene inspects the screen as a human tester would to judge whether the expected result is visible. Describe the expected appearance in natural language to check colors, selection highlights, layout, and visual feedback — including content drawn on <canvas> or displayed in native apps.

await agent.aiAssert('The selected plan has a blue border and a checkmark');
await agent.aiAssert('The error message is visible below the email field');

Benchmark performance

Benchmark Pass@1 Model used in the reported run
AndroidWorld 93.1% Gemini-3.5-Flash
MobileWorld 78.6% Gemini-3.6-Flash
AppControlBench 96.7% Doubao Seed 2.1 Turbo

Each report includes the run configuration and task results; see AndroidWorld's report for its environment and validator adjustments.

Cost and model choice

Screenshot-based UI actions avoid sending large DOM trees to the model. In the reported AppControlBench run, Midscene with Doubao Seed 2.1 Turbo completed the 60-task evaluation with $0.59 in total model cost, achieving 58 passes. The report provides per-task costs and comparisons across models.

Midscene supports multimodal models such as Qwen3.x, Doubao-Seed-2.1, GLM-4.6V, gemini-3.5-flash, and UI-TARS, including open-source options you can self-host. Start with one model, or combine planning and vision models for your workload. For data extraction and page understanding, you can opt in to include DOM. See Model Strategy.

Showcases

🧰 Testing Kit

Batteries included: Midscene provides the test framework, observability, and integration APIs needed to turn GUI automation into a maintainable E2E test project.

Midscene Test: an E2E framework for the AI era

Midscene Test (@midscene/test, Beta) separates declarative test intent from programmable engineering. Write UI flows and expectations in YAML, and wrap API calls, data preparation, and cleanup in reusable TypeScript Nodes. A refund test can prepare an order through an API, request a refund through the UI, and verify the result in one workflow.

The framework includes project scaffolding, platform presets, lifecycle hooks, retries, and execution-project isolation and concurrency. It also generates a Markdown reference from registered Nodes and their parameter schemas, so people and AI Agents can discover the same capabilities and co-maintain test cases. See Create and extend a project and Write and run tests.

Built-in observability

Interactive HTML reports show screenshots, element locations, the AI decision process, and action and assertion results. Midscene Test records the inputs, outputs, duration, and status of each AI step and custom business operation. Reports and runtime logs give both developers and AI Agents the context to investigate failures. Use the Playground to try and refine instructions against your interface.

Rich APIs that fit your existing stack

Use aiAct for autonomous flows, aiTap and aiInput for individual actions, aiAssert for assertions, and aiQuery for structured data extraction. Combine these Agent APIs with your existing code, fixtures, and assertions through Playwright, Puppeteer, or the JavaScript SDK. You can adopt Midscene's visual capabilities within your current testing framework. AI coding agents can also operate interfaces through Midscene Skills.

🚀 Get started

📄 Resources

🤝 Community

🌟 Awesome Midscene

Community projects that extend Midscene.js capabilities:

📝 Credits

We would like to thank the following projects:

📖 Citation

If you use Midscene.js in your research or project, please cite:

@software{Midscene.js,
  author = {Xiao Zhou, Tao Yu, YiBing Lin},
  title = {Midscene.js: GUI Agent for E2E Testing.},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/web-infra-dev/midscene}
}

✨ Star History

Star History Chart

📝 License

Midscene.js is MIT licensed.

If this project helps you or inspires you, please give us a star

midscene (web-infra-dev/midscene)

Searching Database…

15.3K ·GitHub
Midscene
What is this?Midscene is a vision-driven GUI Agent combined with an E2E testing kit: it looks at screenshots to act on and verify UI across web, mobile, and desktop apps through one set of Agent APIs.
Is this a Playwright plugin, or a standalone agent?It's its own Agent framework (aiAct, aiTap, aiInput, aiAssert, aiQuery) that you plug into Playwright, Puppeteer, or the JS SDK, or drive declaratively via YAML in Midscene Test — not a plain Playwright plugin.
How do I start writing tests with it?Try it in the Playground or the Chrome extension first, then write tests with the Playwright/Puppeteer SDK or in YAML with Midscene Test, configuring a model and pointing it at an existing Playwright page.
Do I need to bring my own AI model?Yes — you configure a multimodal model such as Qwen3.x, Doubao-Seed-2.1, GLM-4.6V, gemini-3.5-flash, or UI-TARS, and open-source options can be self-hosted instead of using a paid API.
Does it rely on the page DOM, like Selenium selectors do?No — actions and assertions are primarily screenshot-based to avoid sending large DOM trees to the model; you can opt in to include DOM data for extraction tasks.
Can it test mobile and desktop apps too, not just web?Yes — the same Agent APIs work across Web, Android, iOS, HarmonyOS, and desktop apps, with community SDKs extending it to iOS Mirror, Windows/macOS/Linux PC control (including a Docker image), Python, and Java.
What core APIs does Midscene expose?aiAct drives autonomous multi-step flows, aiTap and aiInput perform single actions, aiAssert checks visual expectations, and aiQuery extracts structured data from the screen.
How good is it on real benchmarks?Reported Pass@1 scores include 93.1% on AndroidWorld (Gemini-3.5-Flash), 78.6% on MobileWorld (Gemini-3.6-Flash), and 96.7% on AppControlBench (Doubao Seed 2.1 Turbo), with a full 60-task AppControlBench run costing about $0.59 in model fees.
Can AI coding agents drive the UI with Midscene too?Yes, Midscene Skills let AI coding agents operate interfaces using the same visual Agent capabilities, not just human-written test scripts.
What is Midscene Test?It's a Beta E2E framework that separates YAML-written UI flows and expectations from reusable TypeScript Nodes for API calls, data prep, and cleanup, with scaffolding, retries, and auto-generated docs of available Node capabilities.
What are similar projects?mobile-mcp, testsprite-cli, UI-TARS-desktop, e2e, typesafe-computer-use, agent-desktop, collab-agent, playwright
Which topics is it in?gui-agent, computer-use-agent, mobile-automation, web-automation, qa, browser-use, desktop-automation, e2e-testing, gui-automation, test-automation, computer-use, yaml
Creator
web-infra-dev
Size
764.3 MB
Stars
15.3K
Updated
CommitOct 16Releasev1.16.0Oct 16
Age
2y
Commits · 90 Days