ClawBench README

项目的 README,来自 github.com/TIGER-AI-Lab/ClawBench。合并为一个 markdown 文件:README.md。

ClawBench

Can AI agents complete everyday online tasks?

Project Page Paper Leaderboard Dataset

Quick start · Results · Docs · Citation · 中文 · WebsiteBench

ClawBench logo: a paw-shaped popsicle on a weight bench

ClawBench is an open-source benchmark, published in Findings of EMNLP 2026, that evaluates AI browser agents on everyday online tasks — booking travel, ordering food, applying for jobs, managing email — across live websites. It records five layers of evidence and supports request-based scoring and trajectory evaluation against human references. In the historical V1 paper evaluation, the strongest evaluated agent completed about 1 in 3 tasks.

V1: 152 tasks · 143 live sites  |  V2: 129 tasks · 63 live sites  |  281 tasks across 163 live websites · 15 life categories

Built by NAIL Group · Sister project: WebsiteBench — Evaluation of agents' ability in reverse-engineering websites · Runs on any Chrome.

Featured in 37 curated lists and #3 Paper of the Day on Hugging Face: docs/featured.md.


What are you looking for?

I want to… Start here
Run my model Quick start
See scores Live leaderboard · Results
Browse the tasks Task explorer · Datasets
Reproduce a score Reproduction workflow · Trace cookbook
Understand the benchmark Paper · FAQ · How ClawBench compares
Change the framework or add tasks Source layout · Contributing

All guides and integrations: Documentation index.

Quick start

Prerequisites: Python 3.11+, uv, and a running Docker or Podman engine. See platform setup.

Installed package

Install clawbench-eval from PyPI:

uv tool install clawbench-eval
clawbench

Launch from the directory where you want your configuration and results. The interactive TUI guides you through adding a model and selecting a task. Configure the default judge, deepseek-v4-pro, as well; the model setup guide explains judge selection and how to run without scoring.

Source checkout

git clone https://github.com/TIGER-AI-Lab/ClawBench.git
cd ClawBench
cp models/models.example.yaml models/models.yaml
# Edit models/models.yaml with your model and judge API settings.
uv run clawbench

./run.sh is a shortcut for the same TUI. For a non-interactive run from the checkout:

uv run clawbench-run test-cases/v1/886-entertainment-hobbies-experience-topgolf your-provider-model-id

Replace your-provider-model-id with a configured model key. The first run builds container images; the local browser's printed noVNC URL lets you watch the agent. Recordings and metadata land under test-output/.

Continue with the getting started guide for batch and human runs, harness selection, and remote browser options. The CLI reference covers non-interactive commands. Coding agents can start with AGENTS.md.

How it works

   You pick a task            ClawBench spins up           Agent drives the         Interceptor captures
   from V1 or V2              an isolated Docker           browser: navigates,      every action across
   everyday scenarios         container + Chromium         fills forms, clicks      all 5 layers of data

   ┌──────────────┐           ┌──────────────┐           ┌──────────────┐           ┌──────────────┐
   │  "Book a pet │    ──►    │   Container  │    ──►    │   AI Agent   │    ──►    │   5 layers   │
   │   sitter on  │           │  + Chromium  │           │  browses the │           │  intercepted │
   │   Rover"     │           │  + Agent     │           │   live site  │           │  & recorded  │
   └──────────────┘           └──────────────┘           └──────────────┘           └──────────────┘

Live websites · isolated containers · request interception · five-layer recording.

Container internals
┌─────────────────────────────────────────────────┐
│  Container (Docker / Podman)                    │
│                                                 │
│  ┌──────────┐  CDP Fetch/Runtime/Page events    │
│  │ Chromium ├─────────────────────────────┐     │
│  │ :9222 CDP│                             │     │
│  └──────────┘                             │     │
│                                           │     │
│  ┌──────────┐            ┌────────────────▼─┐   │
│  │  Xvfb    │◄──ffmpeg──►│  FastAPI Server  │   │
│  │ :99      │  x11grab   │  :7878           │   │
│  └──────────┘            └──────────────────┘   │
│                                  │              │
│                          ┌───────▼─────────┐    │
│                          │     /data       │    │
│                          │  actions.jsonl  │    │
│                          │  requests.jsonl │    │
│                          │  screenshots/   │    │
│                          │  recording.mp4  │    │
│                          └─────────────────┘    │
└─────────────────────────────────────────────────┘

Datasets

ClawBench ships three Hugging Face datasets — task definitions plus full execution traces for V1 and V2. All open, downloadable in one command.

Dataset What's in it Get it
NAIL-Group/ClawBench (mirrored at TIGER-Lab/ClawBench) Task definitions, rubrics, and metadata for V1 and V2 — what to attempt and how it's judged. hf download --repo-type dataset NAIL-Group/ClawBench
NAIL-Group/ClawBenchV1Trace One directory per V1 model run: recording.mp4, requests.jsonl, actions.jsonl, agent-messages.jsonl, interception.json, run-meta.json. hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace
TIGER-Lab/ClawBenchV2Trace Same 5-layer bundle for V2 runs. Rolling — new models added as they're evaluated. hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace

Build with the traces: See the Trace Cookbook for download recipes, data schemas, and research directions.

The trace datasets are large; use hf download --include "<pattern>" to pull a single model or a single task.

Live leaderboard: claw-bench.com/leaderboard (V2 default, two-stage scoring — interception + LLM judge). Full scoring formula in eval/scoring.md. Add your run: PR to leaderboard/results.csv.

News

Earlier updates are in docs/news.md; the full change history is in CHANGELOG.md.

Awesome Works using ClawBench

Authors from Google DeepMind, Stanford, UC Berkeley, Google, Microsoft Research, Harvard, ETH Zürich, Oxford, Northwestern, ByteDance Seed, and HKUST build on ClawBench — and Li Auto's Mach-Mind-4-Flash technical report evaluates on it.

9 more works

If we missed your work, please open an issue or submit a pull request.

Results

V2 · Hermes harness · 8 models · deepseek-v4-pro judge, lenient and strict rubrics · live at claw-bench.com

Rank Model Harness Intercepted Reward (lenient) Reward (strict) Pass / Total
1 claude-opus-4-7 hermes 54.6% 44.6% 24.6% 58 / 130
2 gpt-5.5 hermes 45.4% 35.4% 18.5% 46 / 130
3 glm-5.1 hermes 48.5% 34.6% 17.7% 45 / 130
4 deepseek-v4-pro hermes 43.9% 33.9% 12.3% 44 / 130
5 openrouter-owl-alpha hermes 14.6% 0.0% 0.0% 0 / 130
6 z-ai/glm-4.5-air:free hermes 4.6% 2.3% 0.8% 3 / 130
7 deepseek-v4-flash:free hermes 3.1% 2.3% 0.0% 3 / 129
8 minimax-m2.5:free hermes 2.3% 1.5% 0.0% 2 / 130

Intercepted = final HTTP request matched the task's URL/method (Stage 1, deterministic). Reward (lenient) = additionally judged by deepseek/deepseek-v4-pro to fulfill the instruction under the "no contradiction → match" rubric (Stage 2). Reward (strict) = same judge, strict rubric ("ambiguous → mismatch"). Ranked by Intercepted; Reward as tiebreak. Totals reflect the corpus size at run time.

Paper results · V1 · trajectory evaluation against human references · Findings of EMNLP 2026

Rank Model Overall
1 Claude Sonnet 4.6 33.3%
2 GLM-5 24.2%
3 Gemini 3 Flash 19.0%
4 Claude Haiku 4.5 18.3%
5 GPT-5.4 6.5%
6 Gemini 3.1 Flash Lite 3.3%

More results are in docs/results.md: the per-category breakdown of the paper results, V2 under OpenClaw, a V1 run under the Hermes harness, and the V1 task categories. Codex and Claude Code runs on V2, and the V1 OpenClaw aggregate, are still in flight — they land on the live leaderboard first.

Reproduce the leaderboard

Two independent runs of the same model under the same judge reproduce Intercepted and Reward within ±2 pp on V2. To check a published row against our traces:

clawbench-reproduce --model deepseek-v4-flash --tolerance 2.0

Re-running the agent, re-judging published traces, and the pass criterion are in docs/reproduce.md.

ClawBench-Lite

New here? Run this first. test-cases/v1-lite/ is a 20-task curated subset of V1, selected for household-name sites, real-world relevance, difficulty, and category diversity. It matches the 20-tasks-per-source convention of browser-use/benchmark and gives you a credible signal at a fraction of the full-benchmark cost.

Tier distribution: flagship 9 / core 8 / wildcard 3 — spanning daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), dev-tech (GitHub), academia (Overleaf), personal management (1Password), and more. Use the separate human-reference evaluation workflow to evaluate Lite trajectories under eval/agentic_eval.md; selecting the suite does not switch the native request judge to trajectory evaluation.

Run it with --cases-suite v1-lite, or inspect the link-backed task files in test-cases/v1-lite/.

Example walkthrough

The task walkthrough follows a V1 task from its JSON specification through the recorded browser session and human-reference assessment. It also explains why a placeholder interception rule needs trajectory evaluation.

Evaluation

The native runner scores intercepted requests with an LLM judge by default: a pass requires both an interception and a matching judge verdict. --no-judge reports interception-only results; clawbench-rescore can judge saved requests later. See the CLI reference.

The original V1 evaluation compares complete agent and human-reference trajectories under eval/agentic_eval.md. This is a separate workflow described in the evaluation guide. Compare scores using the same corpus revision, harness, and rubric.

To share rescored results, see the OpenEval / EvalPort export instructions.

CLI

./run.sh                                                                   # interactive TUI
uv run clawbench-run <case-dir> <model>                                    # one task
uv run clawbench-run <case-dir> --human                                    # human reference run
uv run clawbench-batch --models <model> --cases-suite v2 --all-cases       # a whole corpus

Command overview and commonly used flags: CLI reference. Run any command with --help for its complete option list.

V1 tasks are in test-cases/v1/ (152 tasks), V2 in test-cases/v2/ (129), Lite in test-cases/v1-lite/ (20), and converted Claw-Eval tasks in test-cases/claw-eval/ (19). All suites use test-cases/task.schema.json. For test case authoring, see CONTRIBUTING.md; for output structure and evaluation guidance, see eval/README.md.

How ClawBench compares

ClawBench's niche: live consumer websites, everyday tasks, end-to-end recording. If you want a controlled sandbox or replayed traces, benchmarks such as WebArena and Mind2Web are excellent. If you want to know whether your agent can actually order food or book a flight today, this is the benchmark for that. The side-by-side table with WebArena, GAIA, SWE-bench, Mind2Web and five other benchmarks is in docs/comparison.md.

FAQ

See the FAQ for task coverage, account setup, recording artifacts, scoring, and live-site limitations. For installation and model configuration, use Getting started.

Contributing

We welcome contributions -- especially new test cases. If you've ever ordered groceries, booked an appointment, or filed a form online, you already know how to write one. Most PRs are a single JSON file and land in under a day.

Quick wins:

See CONTRIBUTING.md for the full guide and contributor recognition policy.

Community

Citation

If you use ClawBench in your research, please cite:

@inproceedings{zhang2026clawbench,
  title={Clawbench: Can ai agents complete everyday online tasks?},
  author={Zhang, Yuxuan and Wang, Yubo and Zhu, Yipeng and Du, Penghui and Miao, Junwen and Lu, Xuan and Li, Zhuofeng and Qu, Xingwei and Guo, Zhengkang and Shen, Yuanzhe and others},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2026},
  pages={9386--9408},
  year={2026}
}

Contact

For questions, suggestions, bug reports, or feature requests, please open a GitHub issue.

Core Contributors


Yuxuan Zhang

Yubo Wang

Perry Zhu

Penghui Du

Junwen Miao

Advisors


Kelsey R. Allen

Wenhu Chen

Dongfu Jiang

Liang Chen

Support ClawBench

If ClawBench is useful for your research or product work, the single most helpful thing you can do is star the repo — it surfaces the benchmark to other AI-agent researchers and helps us justify continued dataset curation.

Open to contributions — new test cases, bug fixes, or evaluation submissions for a model we haven't scored yet. See CONTRIBUTING.md.

Contributors

Star History

Star History Chart

License & Acknowledgments

Apache 2.0 -- see LICENSE.

The converted Claw-Eval suite in test-cases/claw-eval/ is derived from claw-eval/claw-eval and the claw-eval/Claw-Eval dataset, which are released under the MIT License. Third-party package notices are in NOTICE.

Built with OpenClaw, opencode, Claude Code, the Claude in Chrome extension, OpenAI Codex CLI, browser-use, claw-code, Hermes Agent, Pi with pi-browser-harness, and WebBrain (selectable harnesses), Microsoft Playwright MCP (browser control bridge for the opencode, claude-code, codex, and claw-code harnesses), LiteLLM (API translation proxy for the claude-code, claude-code-chrome-extension, codex, browser-use, claw-code, and pi harnesses), noVNC (MPL 2.0), and websockify (LGPL 3.0).

ClawBench (TIGER-AI-Lab/ClawBench)

正在搜索数据库…

这是什么?ClawBench 是一个开源基准测试,用于评估 AI 浏览器智能体在日常网上任务中的表现,例如订机票酒店、点外卖、申请工作、管理邮件,所有测试都在真实在线网站上进行。
这是我运行的 AI 智能体,还是用来给智能体打分的测试套件?它是一个基准测试框架,本身并不是智能体。你接入一个模型或编程智能体,它会在真实网站上操控浏览器,ClawBench 再用自带的评判模型记录并给这次运行打分。
我怎样用自己的模型运行评测?在工作目录中安装 ClawBench 软件包,然后启动其交互式 TUI,添加模型并选择任务;也可以克隆源代码,配置好模型密钥后运行 ./run.sh,进行非交互式运行。
运行 ClawBench 之前需要安装什么?Python 3.11+、uv 包管理器,以及正在运行的 Docker 或 Podman 引擎(任务在隔离的容器中运行,你可以通过 noVNC 观看浏览器的操作)。
目前的智能体在这些任务上表现如何?并不理想:在 V1 的原始论文评测中,表现最好的前沿智能体只完成了约三分之一的日常任务;在 V2 更严格的评判标准下,顶尖模型的通过率远低于 25%。
究竟有哪些人在使用 ClawBench?主要是对浏览器/电脑操作类智能体进行基准测试的 AI 实验室和学术研究人员。贡献者和论文都会引用基于它开展工作的团队,包括 Google DeepMind、斯坦福大学、加州大学伯克利分校、Microsoft Research、苏黎世联邦理工学院等。
ClawBench 在 Hugging Face 上发布了哪些数据集?共三个数据集:V1 和 V2 的任务定义与评分标准,以及两个版本各自独立的完整执行轨迹包(包括视频、请求、操作和智能体消息),均可通过 hf download 下载。
智能体的一次运行具体是如何打分的?评分分两个阶段:第一阶段检查智能体最后一次 HTTP 请求是否与任务的目标 URL 和方法完全一致;第二阶段由评判模型(deepseek-v4-pro)按宽松或严格的标准评定任务是否完成。
这个基准测试的规模和覆盖范围有多大?V1 和 V2 合计共 281 个任务,涵盖 163 个真实在线网站,分布在 15 个日常生活类别中,如旅行、金融、求职和宠物护理等。
我可以评测哪些智能体框架?内置支持 Hermes 和 OpenClaw 框架,以及 Browserbase、Kernel 等远程浏览器运行时;近期更新还新增了 WebBrain 适配器,以及 EdgeBench/SForge 集成。
大小
5.4 MB
星标
1.3K
时长
7个月
提交 · 90 天