WebsiteBench README

项目的 README,来自 github.com/TIGER-AI-Lab/WebsiteBench。合并为一个 markdown 文件:README.md。

WebsiteBench

Can AI agents rebuild websites through browser exploration?

Project Page Leaderboard Tasks Harbor Hub Paper

Quickstart · Tasks · Run in the cloud · Docs · PyPI · Citation

Given a production-level website they can explore through a browser, agents must reverse-engineer a full-stack version that matches its appearance and behavior as closely as possible.

Tasks

70 websites and 46,916 hidden test cases, on Harbor Hub and Hugging Face. See the task list.

Download them with the PyPI package:

pip install websitebench
websitebench download --task medium

Running the benchmark

Install Docker and Harbor:

uv tool install harbor==0.25.0

Run an agent on all 70 tasks:

export OPENAI_API_KEY='<key>'
harbor run --dataset websitebench/WebsiteBench@v1 \
  --agent codex --model '<model>' --n-concurrent 1 \
  --allow-environment-host snapshot.debian.org \
  --allow-environment-host raw.githubusercontent.com \
  --allow-environment-host github.com \
  --allow-environment-host nodejs.org \
  --allow-environment-host registry.npmjs.org

To run one task, replace --dataset websitebench/WebsiteBench@v1 with --task websitebench/medium@v1.

Check a task with its reference solution:

harbor run --task websitebench/medium@v1 --agent oracle

Each trial's reward is in jobs/<job>/<trial>/verifier/reward.txt. The docs cover other agents, downloads and cloud machines.

Scoring

Each task has 302 to 2,016 hidden test cases. A case passes when its functional checks pass and every visual checkpoint reaches a normalized SSIM of 0.99. A task's reward is the share of its cases that pass.

Citation

@misc{zhang2026websitebench,
  title  = {WebsiteBench: Can AI Agents Rebuild Websites through Browser Exploration?},
  author = {{WebsiteBench Authors}},
  year   = {2026},
  url    = {https://github.com/TIGER-AI-Lab/WebsiteBench},
}

License

The benchmark code, task definitions, and verifier are released under Apache-2.0. The reference websites reproduce third-party sites for evaluation; their content belongs to its owners and is provided for non-commercial research only.

The task definitions carry a harbor-canary GUID. Please exclude documents that contain it from training data.

Star History

Star History Chart

WebsiteBench (TIGER-AI-Lab/WebsiteBench)

正在搜索数据库…

这是什么?一个基准测试:AI 智能体在浏览器中探索一个正在线上运行的生产级网站,并需要逆向还原出一个外观和行为都与之一致的全栈副本。
这是一个需要安装的工具,还是一个需要下载的数据集?两者都是:70 个任务及其隐藏测试用例来自 Harbor Hub 或 Hugging Face,你需要借助 Docker,通过 Harbor 在这些任务上运行智能体。
我该怎么让智能体在它上面运行?先安装 Docker 和 Harbor,然后在全部 70 个任务上运行你的智能体。若只想试一个任务,把数据集参数替换为 --task websitebench/medium@v1 即可。
开始之前我需要准备什么?Docker 和 Harbor。任务可以通过 PyPI 包下载,也可以从 Harbor Hub 和 Hugging Face 获取。
我能在云服务器上运行它,而不是用自己的电脑吗?可以,文档中有关于云服务器的内容,但 README 没有说明如何搭建,也没有说明费用。
参考网站能用于商业用途吗?不能。代码、任务和验证器采用 Apache-2.0 许可,但参考网站归其所有者所有,仅可用于非商业研究。
这个基准测试到底包含什么?70 个网站,每个网站有 302 至 2,016 个隐藏测试用例,共计 46,916 个用例。每个网站都会按照大量功能和视觉检查项进行评分。
在完整运行之前,我能只测试一个网站吗?可以。把数据集参数替换为 --task websitebench/medium@v1 即可只运行一个任务,也可以先对照参考答案检查任意任务。
它评分时会同时看外观和行为吗?会。每个用例都把功能检查与通过 SSIM 比对的视觉检查点结合起来,因此智能体既要还原网站的运作方式,也要还原它的外观。
大小
4.6 MB
星标
511
更新
提交Nov 10
时长
14天
提交 · 90 天