▶ Watch the launch video — the three hard limits every coding agent hits, and the loop that breaks them (4 min).
What is it?
TestSprite is the AI testing platform 100,000+ teams use to test their software, frontend and backend — in the cloud, against the live product, not mocks. This repo is its official CLI.
It puts that platform in your coding agent's hands: the agent verifies every behavior it ships, and what broke comes back as one self-consistent bundle it can act on — no dashboard scraping. Humans drive the same surface from a terminal or CI.
⭐️ Star the Repository
If you find testsprite useful, a GitHub Star ⭐️ would be greatly appreciated — it helps other builders (and their agents) find the project, and stars notify you about new releases.
Quickstart
Requires Node.js 20.19+, 22.13+, or 24+. (No global install? npx @testsprite/testsprite-cli works too.)
npm install -g @testsprite/testsprite-cli
testsprite setup
testsprite setup prompts for your API key, verifies it, and installs the verification-loop skill for the coding agents this project uses — one command, so your agent is wired to verify its own work.
Which agent it installs for. Each of the eight supported targets — claude, cursor, cline, windsurf, antigravity, kiro, copilot, codex — lands its skill in a different place, so setup works out which you use: the agent calling it, if it identifies itself, together with every agent whose config is already in the project. It installs for all of them and prints what it found — in a terminal it shows you that list first, so you can accept it or narrow it before anything is written. If nothing identifies an agent, it installs for claude and says so — that is the one case where you want --agent <target> to name one yourself. Check the result any time with testsprite agent status.
Non-interactive (CI / onboarding scripts):
TESTSPRITE_API_KEY=sk-... testsprite setup --from-env --yes
Pointing a coding agent (Claude Code, Cursor, Codex, Cline, …) at TestSprite? Have it run testsprite setup first — that installs the verification skill, so the agent knows how to create, run, and triage tests on its own (instead of guessing from this README). setup reports which agents it installed for; add --agent <target> if that list is not the one you wanted. New here? Start with the getting-started overview.
Privacy note: interactive runs check the npm registry at most once per 24 h to offer a "new version available" notice — package name only, never your key or data; TESTSPRITE_NO_UPDATE_NOTIFIER=1 disables it. The backend also advertises its minimum supported CLI version — a below-floor CLI prints a one-line upgrade advisory on stderr, and a too-old client may be rejected with exit 14 (CLIENT_TOO_OLD). Details in DOCUMENTATION.md → Update notice.
From there, the loop runs on its own — an example session, typed by the coding agent:
# 1 — describe the behavior you want to guarantee, run it, wait
testsprite test create --project proj_8f0f6 --type frontend \
--plan-from ./checkout-flow.plan.json --run --wait --output json
# → exits 1: the run failed
# 2 — pull ONE self-consistent failure bundle
testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure
# 3 — the agent reads the bundle, fixes the code, then replays
testsprite test rerun test_3a9f21c7 --wait --output json
# → exits 0: passed. The test now lives in your durable suite.
./checkout-flow.plan.json (the --plan-from argument above) is a JSON file describing the test in plain language — no browser code required. This is the exact, byte-identical output of testsprite test create --plan-template (also embedded verbatim in test create --help):
{
"$schema": "https://raw.githubusercontent.com/TestSprite/testsprite-cli/v0.4.0/schemas/plan.schema.json",
"projectId": "prj_abc123",
"type": "frontend",
"name": "Login rejects an empty password",
"planSteps": [
{
"type": "action",
"description": "Navigate to /login and submit the form with an empty password"
},
{
"type": "assertion",
"description": "Verify an inline error says the password is required"
}
]
}
Get this exact skeleton without hand-copying it (and without the risk of it drifting from your installed version — see below): testsprite test create --plan-template. Full field reference (including the {{...}}-placeholder caveat, size caps, and the $schema hook for live editor validation): Plan file format.
The $schema URL above is pinned to the CLI version that generated this doc (v0.4.0) — --plan-template's live output always pins to your installed version instead, which is what actually resolves. If you're reading this on a later release, run the command yourself rather than trusting this snippet verbatim.
Don't want to write the first tests yourself? For a deployed target, ask TestSprite to propose them. Local projects use the authored-plan flow below. testsprite test plan generate --project <id> explores your app (or reads the API spec you uploaded with project docs upload), drafts test cases, and stages them for review — nothing is written to your disk. You read the printed table and keep what you want:
testsprite test plan generate --project proj_8f0f6 # stages proposals for review
testsprite test plan accept --project proj_8f0f6 # → real tests
testsprite test plan accept --project proj_8f0f6 --only prop_2 prop_5 # or just these
It only runs what the project is still missing, so a second call picks up where the first left off. Generation spends workspace credits per stage that actually runs (the result line reports the spend), and a PRD uploaded with project docs upload --role prd bills 0.5 credits for its embedding — an API spec is free. Full walkthrough, including what happens on Ctrl-C and every precondition error: Generate commands.
Prefer to configure each step by hand (or learn the surface offline with --dry-run first)? See Manual setup and Install & verify.
Testing an app on your machine (--local)
Frontend tests can reach your running app through a TestSprite tunnel, including on the Free plan (ordinary run credits apply).
Use an API key with run:tunnel: keys minted before that scope existed need to be replaced; the CLI names the missing scope.
testsprite project create --type frontend --name "My local app" --local 3000
testsprite test create --plan-from ./checkout.plan.json --project <project-id>
testsprite test run <test-id> --local 3000 # waits up to 1200 seconds
testsprite test run <id> <id> --local 3000 --max-concurrency 5
testsprite test run --all --project <project-id> --local 3000
# Or keep one tunnel open in terminal A (no port argument)
testsprite tunnel start --ttl 3600
# Terminal B: borrow the clientId printed above; keep terminal A running
testsprite test run <test-id> --local 3000 --tunnel-client <client-uuid>
testsprite tunnel list
testsprite tunnel stop <client-uuid>
# Stop every live tunnel on this account only when you intend to revoke them all
testsprite tunnel stop --all --confirm
Run several local frontend tests in one invocation to share one tunnel binding. The batch runs up to 5 tests at once by default; --max-concurrency accepts 1–10. --all --local skips backend tests. An owned batch cancels unfinished runs on Ctrl-C or tunnel loss; JSON output lists every requested test in input order, including tests that did not start. The minted client id is printed before the tunnel connects.
Hit the tunnel limit? Run testsprite tunnel list, then testsprite tunnel stop <client-uuid> for an unused binding, or testsprite tunnel stop --all --confirm to revoke them all. An older server without listing returns exit 4; a key without run:tunnel returns exit 3.
A second tunnel start or process using the same credential takes over, and the first exits 10.
The examples use 127.0.0.1. For another loopback listener, pass the same --local-host localhost or --local-host ::1 to project creation and every follow-up run. The selected host is stored in the project URL (::1 becomes http://[::1]:<port>); project get/list exposes originMode: 'local' in JSON and shows (Local) in text output.
The control plane is WebSocket over TLS (wss://control.tun.testsprite.com/ws). The data plane, which carries the tunnel secret and proxied traffic, uses TLS at data.tun.testsprite.com:443 and verifies the certificate with Node's default trust store: its bundled Mozilla roots, plus certificates supplied through NODE_EXTRA_CA_CERTS and the system CAs when Node is started with --use-system-ca. Node 20 is supported without losing NODE_EXTRA_CA_CERTS. Verification cannot be disabled, and the CLI never falls back from TLS to plaintext. On a network that re-signs TLS, export your organisation's root CA to a PEM file and set NODE_EXTRA_CA_CERTS=/path/to/ca.pem before running testsprite. Plaintext connects and TLS handshakes each time out after 10 seconds. The first failed attempt starts a 60-second retry episode; writing TunnelHello is not enough to reset it. The episode ends only after the first inbound tunnel stream or after the socket remains open for 5 seconds following the hello. A real deadline timer stays armed across retries and destroys an in-flight socket when it expires. After that, an owned test run --local run is cancelled and refunded and the command exits 10; tunnel start exits 10. If a self-hosted or older TestSprite server does not advertise a TLS endpoint, the CLI prints a one-time warning and uses legacy plaintext port 7400 under the same timeout and retry rules. tunnel start prints transport: tls or transport: plaintext.
An owned tunnel's timeout or Ctrl-C cancels unfinished runs by default. A borrowed run still detaches without cancellation on Ctrl-C/SIGTERM because its owner keeps the tunnel alive; if that owner disappears while a run is active, the borrower cancels its own run and reports cancelled, already finished, or skipped. A run cancelled before it finishes is refunded. --no-cancel-on-interrupt also skips this owner-gone cancellation. Keep the early stderr run-id receipts (Run <runId> for one test, [i/N] <testId> → run <runId> for a batch) and read the reported runs before re-running.
Local projects skip exploration/plan generation; author plans with test create --plan-from. Portal runs are blocked for free until you set a public project URL.
See the full local-testing reference for flags, login, billing/refunds, timeouts, and retargeting.
Commands
| Group |
Command |
What it does |
| Setup |
setup |
Start here — one command: configure your API key, verify it, and install the agent verification skill |
|
doctor |
Environment diagnostic — CLI/Node versions, profile, endpoint, credentials, connectivity, agent skill; exits non-zero on failure |
| Auth |
auth status |
Resolve the active profile to its user, key, env, and scopes |
|
auth remove |
Remove the active profile from the credentials file |
|
usage (alias credits) |
Account pre-flight: identity, plus credit balance / plan info when the backend supplies them |
| Read |
project list / project get |
List projects / fetch one by id |
|
test list / test get |
List tests under a project / fetch one by id |
|
test code get |
Print (or write) the generated test source |
|
test steps |
List the latest run's steps with screenshot / DOM pointers |
|
test result |
Latest result; --history lists a test's prior runs (ENV column; --env <name> filters by environment) |
|
test failure get |
The agent entry point: one self-contained latest-failure bundle |
|
test failure summary |
One-screen triage card (no media download) |
|
test diff |
Compare two runs — verdict, failure kind, per-step status flips, code-version drift |
| Write |
test scaffold / test lint |
Author plans locally: emit a schema-correct starter, validate plan files offline — no network, no credentials |
|
test create / test create-batch |
Create a test (or bulk-create from a plan file); --step-timeout <ms> sets a per-test timeout; dependency flags wire BE metadata |
|
test update / test delete / test delete-batch |
Edit metadata, set/clear the step timeout (--step-timeout / --clear-step-timeout), or permanently delete (--confirm required) |
|
test code put |
Replace generated code (etag-guarded) |
|
test plan put |
Replace a frontend test's plan-steps |
|
project create / project update / project delete |
Manage projects; --test-id-attributes <list> registers the app's own test-hook attributes (priority order) for exported locators, --clear-test-id-attributes reverts; delete removes a project and everything under it (--confirm required, no restore window) |
|
project credential / project auto-auth |
Configure backend-test auth: a static injected credential, or auto-refresh login (Pro) |
|
project env list / create / update / delete / set-default |
Manage a project's environments — named URL + login bundles that test run --env <name> selects. An app on your own machine is named with --local <port> instead of --url — the same spelling as project create — and run with test run --local <port>; delete refuses the default environment |
| Generate |
test plan generate |
Ask TestSprite to propose test cases for a project — proposals are staged server-side for review, never written to disk; only the stages the project is missing actually run |
|
test plan accept |
Turn staged proposals into real tests — all of them, or a subset with --only |
|
project docs upload |
Upload an API spec or PRD as a project source, so generation has something to read |
| Run |
test run |
Trigger a fresh run; --wait blocks until terminal; --all --project <id> runs all tests in a project in wave order --env <name> runs against a named environment. |
|
test rerun |
Replay one/many tests (FE verbatim; BE with deps); billed as a run (0.5 credits FE / 0.2 BE); --all --project <id> reruns all tests --env <name> replays against a named environment. |
|
test flaky |
Replay a test several times (auto-heal off) and report a stability score |
|
test wait |
Block on one or more runIds until terminal |
|
test cancel |
Cancel one or more in-flight runs (Ctrl-C during --wait only detaches, cancel is the real stop — except an owned --local run, which Ctrl-C cancels by default) |
|
test artifact get |
Download the failure bundle for a specific runId |
| Test lists |
testlist list / testlist get |
Inspect saved test lists — named, cross-project collections with per-project environment pins (V3-only) |
|
testlist create / update / add / remove / delete |
Manage a list and its cases; --project-env <id>:<env> pins environments; delete requires --confirm |
|
testlist run |
Run every case (or a --case subset) through its pinned environment; --wait polls all runs, --report junit writes a CI sidecar |
| Schedule |
schedule list / schedule get |
List schedules visible to the key / fetch one by id (status is ENABLED, PAUSED, or AUTO_PAUSED after repeated failures) |
|
schedule create |
Run a project or test list on a 5-field cron; states the frequency before sending and the per-run / monthly credit cost once priced |
|
schedule update |
Retime or rename a schedule, change its recipients, --pause / --resume it |
|
schedule delete |
Remove a schedule and its run history, stopping future triggers (--confirm required, no restore window) |
|
schedule run list |
A schedule's past runs, with per-run pass / fail / blocked counts |
| Agent |
agent install / agent list / agent status |
Add, list, or health-check coding-agent skills (pure-local): claude, codex, cursor, cline, antigravity, kiro, windsurf, copilot |
The earlier command names — init, auth configure, auth whoami, auth logout — still work as hidden, deprecated aliases (each prints a one-line notice pointing at the new name), so existing scripts keep running. auth configure now runs the full setup (it also installs the skill).
📚 Full reference — every command, flag, and example: DOCUMENTATION.md, including configuration & profiles, scripting, and exit codes.
Why a CLI for coding agents?
- 🧪 Tests like a real user. Runs against a live browser or API in the cloud — real clicks, real navigation, real assertions. Not a mock.
- 🤖 Agent-shaped output.
test failure get returns one bundle — the failing step, its neighbors, screenshots, DOM snapshots, the test source, a root-cause hypothesis, and a recommended fix target — all sharing a single snapshotId. The CLI refuses to stitch data from two different runs, so an agent never reasons over a frankenstein context.
- ♻️ A loop, not a one-shot.
create → run → failure get → fix → rerun — every pass is banked, not thrown away.
- 📐 Scriptable & deterministic. Stable
--output json contract, predictable exit codes, and a --dry-run that exercises the full code path offline with canned data.
- 🚦 CI-native. On GitHub Actions,
--wait runs annotate the PR checks tab with one ::error:: per failure and append a results table to the job summary — automatically. Add --report junit for a JUnit XML sidecar, --summary-file for a machine summary, or --gh-output to preview the annotations locally. Runs in any CI, not just GitHub. Details →
- 🔌 One command to onboard your agent.
testsprite setup drops ready-made skill files into your repo — for the agents it detects you using — so your coding agent knows how to drive the loop on its own.
How it works
Every time your agent changes code, it asks one question: is this behavior already covered by the suite?
- Not yet covered →
testsprite test create — describe the new behavior, run it.
- Already covered →
testsprite test rerun — replay the existing tests, so nothing that used to work breaks silently.
- Something fails →
testsprite test failure get — one self-consistent bundle; the agent fixes the code and reruns.
Every pass is banked into a durable suite, so coverage compounds as the project grows — a lasting record of every requirement it has ever gotten right, far bigger than any context window.
flowchart TD
A["🤖 Your coding agent<br/>Claude Code · Codex · Antigravity · Kimi · Cursor · Trae …"]
D{"behavior already<br/>covered by the suite?"}
B["<b>testsprite test create</b><br/>new behavior → new test"]
R["<b>testsprite test rerun</b><br/>replay the existing tests"]
C{{"☁️ TestSprite testing agent<br/>runs the test like a real user against<br/>real browsers & real APIs on Cloud"}}
F["<b>testsprite test failure get</b><br/>ONE self-consistent bundle:<br/>failing step · screenshots · DOM ·<br/>root-cause · recommended fix"]
S[("📚 Durable integration suite<br/>grows with every pass")]
A -->|"writes / changes code"| D
D -->|"no — new behavior"| B
D -->|"yes"| R
B --> C
R --> C
C -->|"pass ✅"| S
C -->|"fail ❌"| F
F -->|"agent reads the bundle<br/>& fixes the code"| A
S -.->|"defines what's covered"| D
The cloud is a black box on purpose: your agent describes intent and reads results. It never has to know how the test was driven — only what a real user experienced.
Proved in public
On CoderCup — an open leaderboard where frontier coding agents build the same app under the same rules, with TestSprite as the referee — the cheapest model in the field shipped the most correct app on the board: 89%, at half the cost of the priciest one.
That's the point of all of this: you no longer need the biggest, most expensive model to ship software you can trust — top-tier quality, without paying top-tier prices, within reach of every team.
Getting help
Contributing
Contributions are welcome — the CLI is plain TypeScript/Node (20.19+, 22.13+, or 24+), tested with Vitest, built with tsc. Getting a dev loop running takes a minute:
git clone https://github.com/TestSprite/testsprite-cli.git
cd testsprite-cli && npm install
npm run build # tsc → dist/ ; try it: node dist/index.js --help
npm test # unit suite — no network or credentials needed
npm run lint:fix # ESLint
npm run typecheck # tsc --noEmit
Pull requests target the main branch. The full guide — build from source, test tiers, CI checks, branches, project layout — is in CONTRIBUTING.md.
Support — if you need any help, we're responsive on Discord, and feel free to email us at [email protected] too.
License
Apache-2.0 © TestSprite
TestSprite 的官方命令行工具。TestSprite 是一个 AI 测试平台,已有超过 10 万个团队用它对前端和后端进行云端测试,测试对象是真实运行的产品而不是模拟数据 —— 发现问题后还会反馈给你的编程智能体,让它自己修复自己写的代码。