项目的 README,来自 github.com/TIGER-AI-Lab/ClawBench。合并为一个 markdown 文件:README.md。
ClawBench is an open-source benchmark, published in Findings of EMNLP 2026, that evaluates AI browser agents on everyday online tasks — booking travel, ordering food, applying for jobs, managing email — across live websites. It records five layers of evidence and supports request-based scoring and trajectory evaluation against human references. In the historical V1 paper evaluation, the strongest evaluated agent completed about 1 in 3 tasks.
V1: 152 tasks · 143 live sites | V2: 129 tasks · 63 live sites | 281 tasks across 163 live websites · 15 life categories
Built by NAIL Group · Sister project: WebsiteBench — Evaluation of agents' ability in reverse-engineering websites · Runs on any Chrome.
Featured in 37 curated lists and #3 Paper of the Day on Hugging Face: docs/featured.md.
| I want to… | Start here |
|---|---|
| Quick start | |
| Live leaderboard · Results | |
| Task explorer · Datasets | |
| Reproduction workflow · Trace cookbook | |
| Paper · FAQ · How ClawBench compares | |
| Source layout · Contributing |
All guides and integrations: Documentation index.
Prerequisites: Python 3.11+, uv, and a running Docker or Podman engine. See platform setup.
Install clawbench-eval from PyPI:
uv tool install clawbench-eval
clawbench
Launch from the directory where you want your configuration and results.
The interactive TUI guides you through adding a model and selecting a task.
Configure the default judge, deepseek-v4-pro, as well; the
model setup guide explains judge
selection and how to run without scoring.
git clone https://github.com/TIGER-AI-Lab/ClawBench.git
cd ClawBench
cp models/models.example.yaml models/models.yaml
# Edit models/models.yaml with your model and judge API settings.
uv run clawbench
./run.sh is a shortcut for the same TUI. For a non-interactive run from the
checkout:
uv run clawbench-run test-cases/v1/886-entertainment-hobbies-experience-topgolf your-provider-model-id
Replace your-provider-model-id with a configured model key. The first run
builds container images; the local browser's printed noVNC URL lets you watch
the agent. Recordings and metadata land under test-output/.
Continue with the getting started guide for
batch and human runs, harness selection,
and remote browser options. The CLI reference covers
non-interactive commands. Coding agents can start with AGENTS.md.
You pick a task ClawBench spins up Agent drives the Interceptor captures
from V1 or V2 an isolated Docker browser: navigates, every action across
everyday scenarios container + Chromium fills forms, clicks all 5 layers of data
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ "Book a pet │ ──► │ Container │ ──► │ AI Agent │ ──► │ 5 layers │
│ sitter on │ │ + Chromium │ │ browses the │ │ intercepted │
│ Rover" │ │ + Agent │ │ live site │ │ & recorded │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
Live websites · isolated containers · request interception · five-layer recording.
┌─────────────────────────────────────────────────┐
│ Container (Docker / Podman) │
│ │
│ ┌──────────┐ CDP Fetch/Runtime/Page events │
│ │ Chromium ├─────────────────────────────┐ │
│ │ :9222 CDP│ │ │
│ └──────────┘ │ │
│ │ │
│ ┌──────────┐ ┌────────────────▼─┐ │
│ │ Xvfb │◄──ffmpeg──►│ FastAPI Server │ │
│ │ :99 │ x11grab │ :7878 │ │
│ └──────────┘ └──────────────────┘ │
│ │ │
│ ┌───────▼─────────┐ │
│ │ /data │ │
│ │ actions.jsonl │ │
│ │ requests.jsonl │ │
│ │ screenshots/ │ │
│ │ recording.mp4 │ │
│ └─────────────────┘ │
└─────────────────────────────────────────────────┘
ClawBench ships three Hugging Face datasets — task definitions plus full execution traces for V1 and V2. All open, downloadable in one command.
| Dataset | What's in it | Get it |
|---|---|---|
| NAIL-Group/ClawBench (mirrored at TIGER-Lab/ClawBench) | Task definitions, rubrics, and metadata for V1 and V2 — what to attempt and how it's judged. | hf download --repo-type dataset NAIL-Group/ClawBench |
| NAIL-Group/ClawBenchV1Trace | One directory per V1 model run: recording.mp4, requests.jsonl, actions.jsonl, agent-messages.jsonl, interception.json, run-meta.json. |
hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace |
| TIGER-Lab/ClawBenchV2Trace | Same 5-layer bundle for V2 runs. Rolling — new models added as they're evaluated. | hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace |
Build with the traces: See the Trace Cookbook for download recipes, data schemas, and research directions.
The trace datasets are large; use
hf download --include "<pattern>"to pull a single model or a single task.
Live leaderboard:
claw-bench.com/leaderboard(V2 default, two-stage scoring — interception + LLM judge). Full scoring formula ineval/scoring.md. Add your run: PR toleaderboard/results.csv.
Earlier updates are in docs/news.md; the full change history is in CHANGELOG.md.
Authors from Google DeepMind, Stanford, UC Berkeley, Google, Microsoft Research, Harvard, ETH Zürich, Oxford, Northwestern, ByteDance Seed, and HKUST build on ClawBench — and Li Auto's Mach-Mind-4-Flash technical report evaluates on it.
If we missed your work, please open an issue or submit a pull request.
V2 · Hermes harness · 8 models · deepseek-v4-pro judge, lenient and strict rubrics · live at claw-bench.com
| Rank | Model | Harness | Intercepted | Reward (lenient) | Reward (strict) | Pass / Total |
|---|---|---|---|---|---|---|
| 1 | claude-opus-4-7 | hermes | 54.6% | 44.6% | 24.6% | 58 / 130 |
| 2 | gpt-5.5 | hermes | 45.4% | 35.4% | 18.5% | 46 / 130 |
| 3 | glm-5.1 | hermes | 48.5% | 34.6% | 17.7% | 45 / 130 |
| 4 | deepseek-v4-pro | hermes | 43.9% | 33.9% | 12.3% | 44 / 130 |
| 5 | openrouter-owl-alpha | hermes | 14.6% | 0.0% | 0.0% | 0 / 130 |
| 6 | z-ai/glm-4.5-air:free | hermes | 4.6% | 2.3% | 0.8% | 3 / 130 |
| 7 | deepseek-v4-flash:free | hermes | 3.1% | 2.3% | 0.0% | 3 / 129 |
| 8 | minimax-m2.5:free | hermes | 2.3% | 1.5% | 0.0% | 2 / 130 |
Intercepted = final HTTP request matched the task's URL/method (Stage 1, deterministic). Reward (lenient) = additionally judged by deepseek/deepseek-v4-pro to fulfill the instruction under the "no contradiction → match" rubric (Stage 2). Reward (strict) = same judge, strict rubric ("ambiguous → mismatch"). Ranked by Intercepted; Reward as tiebreak. Totals reflect the corpus size at run time.
Paper results · V1 · trajectory evaluation against human references · Findings of EMNLP 2026
| Rank | Model | Overall |
|---|---|---|
| 1 | Claude Sonnet 4.6 | 33.3% |
| 2 | GLM-5 | 24.2% |
| 3 | Gemini 3 Flash | 19.0% |
| 4 | Claude Haiku 4.5 | 18.3% |
| 5 | GPT-5.4 | 6.5% |
| 6 | Gemini 3.1 Flash Lite | 3.3% |
More results are in docs/results.md: the per-category breakdown of the paper results, V2 under OpenClaw, a V1 run under the Hermes harness, and the V1 task categories. Codex and Claude Code runs on V2, and the V1 OpenClaw aggregate, are still in flight — they land on the live leaderboard first.
Two independent runs of the same model under the same judge reproduce Intercepted and Reward within ±2 pp on V2. To check a published row against our traces:
clawbench-reproduce --model deepseek-v4-flash --tolerance 2.0
Re-running the agent, re-judging published traces, and the pass criterion are in docs/reproduce.md.
New here? Run this first. test-cases/v1-lite/ is a 20-task curated subset of V1, selected for household-name sites, real-world relevance, difficulty, and category diversity. It matches the 20-tasks-per-source convention of browser-use/benchmark and gives you a credible signal at a fraction of the full-benchmark cost.
Tier distribution: flagship 9 / core 8 / wildcard 3 — spanning daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), dev-tech (GitHub), academia (Overleaf), personal management (1Password), and more. Use the separate human-reference evaluation workflow to evaluate Lite trajectories under eval/agentic_eval.md; selecting the suite does not switch the native request judge to trajectory evaluation.
Run it with --cases-suite v1-lite, or inspect the link-backed task files in test-cases/v1-lite/.
The task walkthrough follows a V1 task from its JSON specification through the recorded browser session and human-reference assessment. It also explains why a placeholder interception rule needs trajectory evaluation.
The native runner scores intercepted requests with an LLM judge by default:
a pass requires both an interception and a matching judge verdict.
--no-judge reports interception-only results; clawbench-rescore can judge
saved requests later. See the CLI reference.
The original V1 evaluation compares complete agent and human-reference
trajectories under eval/agentic_eval.md. This is a
separate workflow described in the evaluation guide.
Compare scores using the same corpus revision, harness, and rubric.
To share rescored results, see the OpenEval / EvalPort export instructions.
./run.sh # interactive TUI
uv run clawbench-run <case-dir> <model> # one task
uv run clawbench-run <case-dir> --human # human reference run
uv run clawbench-batch --models <model> --cases-suite v2 --all-cases # a whole corpus
Command overview and commonly used flags: CLI reference.
Run any command with --help for its complete option list.
V1 tasks are in test-cases/v1/ (152 tasks), V2 in test-cases/v2/ (129), Lite in test-cases/v1-lite/ (20), and converted Claw-Eval tasks in test-cases/claw-eval/ (19). All suites use test-cases/task.schema.json. For test case authoring, see CONTRIBUTING.md; for output structure and evaluation guidance, see eval/README.md.
ClawBench's niche: live consumer websites, everyday tasks, end-to-end recording. If you want a controlled sandbox or replayed traces, benchmarks such as WebArena and Mind2Web are excellent. If you want to know whether your agent can actually order food or book a flight today, this is the benchmark for that. The side-by-side table with WebArena, GAIA, SWE-bench, Mind2Web and five other benchmarks is in docs/comparison.md.
See the FAQ for task coverage, account setup, recording artifacts, scoring, and live-site limitations. For installation and model configuration, use Getting started.
We welcome contributions -- especially new test cases. If you've ever ordered groceries, booked an appointment, or filed a form online, you already know how to write one. Most PRs are a single JSON file and land in under a day.
Quick wins:
See CONTRIBUTING.md for the full guide and contributor recognition policy.
If you use ClawBench in your research, please cite:
@inproceedings{zhang2026clawbench,
title={Clawbench: Can ai agents complete everyday online tasks?},
author={Zhang, Yuxuan and Wang, Yubo and Zhu, Yipeng and Du, Penghui and Miao, Junwen and Lu, Xuan and Li, Zhuofeng and Qu, Xingwei and Guo, Zhengkang and Shen, Yuanzhe and others},
booktitle={Findings of the Association for Computational Linguistics: EMNLP 2026},
pages={9386--9408},
year={2026}
}
For questions, suggestions, bug reports, or feature requests, please open a GitHub issue.
![]() Yuxuan Zhang |
![]() Yubo Wang |
![]() Perry Zhu |
![]() Penghui Du |
![]() Junwen Miao |
![]() Kelsey R. Allen |
![]() Wenhu Chen |
![]() Dongfu Jiang |
![]() Liang Chen |
If ClawBench is useful for your research or product work, the single most helpful thing you can do is star the repo — it surfaces the benchmark to other AI-agent researchers and helps us justify continued dataset curation.
Open to contributions — new test cases, bug fixes, or evaluation submissions for a model we haven't scored yet. See CONTRIBUTING.md.
Apache 2.0 -- see LICENSE.
The converted Claw-Eval suite in test-cases/claw-eval/ is derived from claw-eval/claw-eval and the claw-eval/Claw-Eval dataset, which are released under the MIT License. Third-party package notices are in NOTICE.
Built with OpenClaw, opencode, Claude Code, the Claude in Chrome extension, OpenAI Codex CLI, browser-use, claw-code, Hermes Agent, Pi with pi-browser-harness, and WebBrain (selectable harnesses), Microsoft Playwright MCP (browser control bridge for the opencode, claude-code, codex, and claw-code harnesses), LiteLLM (API translation proxy for the claude-code, claude-code-chrome-extension, codex, browser-use, claw-code, and pi harnesses), noVNC (MPL 2.0), and websockify (LGPL 3.0).