ai-eval-studio README

项目的 README,来自 github.com/awslabs/ai-eval-studio。合并为一个 markdown 文件:README.md。

AI Eval Studio

An end-to-end LLM evaluation platform. Upload a dataset, describe your goal in chat, and the system runs evaluations across models, optimizes the prompt iteratively against real failures, and hands you back a standalone Python eval harness you can run in CI.

Every model call — the agent driving the conversation, the models under test, and the LLM judge — runs on Amazon Bedrock.

This project is a prototype for demonstration, not for production use. Before using it with real data or real users, do your own independent assessment against your security and quality standards. SECURITY.md lists what the default deployment does and does not do, and the steps to take it to production.

What it does

You talk to a coordinator agent, which walks the project through seven phases:

Phase What happens
1. Data analysis Profiles your uploaded CSV/Excel and proposes field mappings
2. Configuration Picks candidate models on a cost-quality frontier and chooses metrics
3. Eval harness Generates a standalone DeepEval harness and validates it runs
4. Baseline prompt Drafts the prompt to optimize from
5. Confirm & run Measures a real baseline across the candidate pool
6. Eval + optimize Iterates prompt edits against failures until scores converge
7. Report Produces a cost/quality report and a recommended model + prompt

The generated harness is the durable artifact: a self-contained Python project that runs without the platform. See backend/app/templates/harness/README.md.

Architecture

The platform runs in two modes from one codebase, selected by EVAL_STUDIO_MODE:

Architecture

Click to view full size. Editable source: architecture-overview.drawio (draw.io).

Components

Running locally

Prerequisites — Python 3.11+, uv, Node.js 22+, git, and AWS credentials.

Anthropic models need a one-time use-case form. Bedrock enables its serverless models automatically, but each account must submit Anthropic's first-time-use form before the first call to a Claude model; submitting it from an organization's management account covers its member accounts. The coordinator agent runs on Claude, so submit the form before the first chat. See Access Amazon Bedrock foundation models.

The local agent runs shell commands on your machine without confirmation. The coordinator writes and executes an evaluation harness, and its Bash tool is not sandboxed or path-restricted — it runs with your user's privileges. Run it on a machine where that is acceptable.

(cd backend  && uv sync)
(cd frontend && npm install)
./scripts/start.sh   # starts backend on :8100 and frontend on :3100
./scripts/stop.sh

Then open http://localhost:3100. start.sh exports CLAUDE_CODE_USE_BEDROCK=1, which the Agent SDK requires.

Both processes log to the terminal and append to .dev-logs/backend.log and .dev-logs/frontend.log (gitignored), so a failure stays readable by someone who wasn't at the keyboard when it happened.

Deploying to AWS

The cloud deployment is AWS CDK (Python) across nine stacks — networking, storage, auth, IAM, observability, compute, frontend, AgentCore, and an AppRegistry application.

The deployed entry point is the CloudFront URL; the ALB is internal and not reachable from the internet. Sign-in is Cognito hosted UI (OIDC + PKCE). The default deployment is a development deployment; SECURITY.md lists the steps to take it to production.

Security scope

Treat one deployment as a single trust domain. Everyone you onboard can reach everyone else's data. The platform is multi-user — Cognito sign-in, per-user projects — and every API route and agent tool call checks ownership before it answers, but an application-layer check is not a boundary the agent itself is confined by: the product generates and runs code, with a shell, by design.

Three specific gaps, named rather than summarised, because "app-layer isolation only" reads as vaguer than the truth:

The second can be closed with AssumeRoleWithWebIdentity, carrying the tenant tag in a claim inside the Cognito ID token so STS verifies it and the caller stops asserting it. The first needs per-session filesystem storage instead of a shared mount; AgentCore offers one and it is in preview. The first is what multi-tenant use actually depends on, and it is not done.

Two related properties are deliberate rather than oversights:

Tests and checks

(cd backend  && uv run pytest)      # backend suite
(cd frontend && npm test)           # frontend suite (vitest)
(cd frontend && npm run build)      # eslint + tsc + vite build
./scripts/lint.sh                   # ruff over backend/, infra/, scripts/
./scripts/test-infra.sh             # CDK synth-level tests (see note)

The infra tests need a runner because they sit between two virtualenvs: they import aws_cdk (installed in infra/.venv) and are collected by pytest (installed in backend/.venv). test-infra.sh puts the former on PYTHONPATH for the latter, so neither venv has to duplicate the other's dependencies.

Tests marked integration hit real external services and are skipped by default; run them with uv run pytest -m integration.

Repository layout

├── frontend/            # React 19 + Vite + TypeScript
├── backend/
│   └── app/
│       ├── api/         # Route handlers (chat, eval, projects, upload, ws)
│       ├── agents/      # Subagent definitions
│       ├── prompts/     # Coordinator + subagent system prompts (markdown)
│       ├── models/      # Pydantic schemas
│       ├── platform/    # dev vs cloud adapters (storage, auth, compute)
│       ├── services/    # Core logic (agent runner, jobs, reports, MCP tools)
│       └── templates/   # Standalone eval harness template
├── infra/               # AWS CDK app (nine stacks)
├── scripts/             # start.sh, stop.sh, lint.sh, test-infra.sh
├── projects/            # User eval projects (gitignored)
└── docs/                # Runbooks, diagrams, reference

Documentation

Doc What it covers
SECURITY.md Reporting a vulnerability; path to production
infra/README.md CDK layout, prerequisites, deploy/iterate loop
docs/runbooks/deploy.md First cloud deploy, step by step
docs/runbooks/teardown.md Full teardown
docs/error-codes.md Stable ResourceError.code catalog
docs/naming-conventions.md File and symbol naming rules
docs/frontend-design-system.md Theming tokens and the no-hardcoded-color rule
docs/diagrams/ Cloud architecture diagram

License

This project is licensed under the Apache-2.0 License. See LICENSE, NOTICE and THIRD-PARTY-LICENSES.

Claude Code

This project depends on the Claude Code CLI, which is proprietary to Anthropic PBC and not covered by this project's license. You must obtain and use it under Anthropic's Commercial Terms of Service.

Reporting a security issue

See SECURITY.md for how to report a potential security issue. Please do not open a public issue.

Known limitations of the deployed system are stated in Security scope above — read that before deploying this for more than one person.

ai-eval-studio (awslabs/ai-eval-studio)

正在搜索数据库…

这是什么?一个端到端的大语言模型评测平台。你上传数据集,在聊天中描述你的目标,它会运行评测、根据真实失败案例优化提示词,并交付一个可独立运行的 Python 评测工具,供 CI 使用。
它是可以导入的库,还是正在运行的应用?是一个正在运行的应用:前端使用 React,后端使用 FastAPI,由一个协调智能体分七个阶段推进,从数据分析一直到最终的成本-质量报告。
如何在本地运行它?安装 Python 3.11+、uv、Node.js 22+、git 和 AWS 凭证,在你的账户中提交 Anthropic 的使用场景表单,然后用 start.sh 启动。日志输出到 .dev-logs。
我需要 Anthropic 的 API 密钥吗?不需要单独的 Anthropic API 密钥,因为所有模型调用都通过 Amazon Bedrock 运行。你需要 AWS 凭证,并且每个账户在首次调用 Claude 之前都必须提交 Anthropic 的首次使用表单。
在自己的电脑上运行安全吗?请注意:本地智能体会在未经确认的情况下执行 shell 命令,而且它的 Bash 工具没有沙箱隔离,因此以你的用户权限运行。
在共享部署中,谁能看到我的数据?你接入的所有人都处于同一个信任域中,因此任何用户都能访问其他用户的数据。项目文件没有按用户隔离,而且已知的漏洞使部分元数据可被跨用户访问,所以只与你信任的人共享。
一切都采用 Apache-2.0 许可证吗?项目代码采用 Apache-2.0 许可证,但它依赖 Claude Code CLI,该工具为 Anthropic 专有,必须遵守 Anthropic 的商业条款使用。
它生成的评测工具是什么?一个自包含的 Python 项目,基于 DeepEval 构建,无需平台即可运行,可以直接放入 CI。它是你最终保留的成果。
每个工作流阶段都做什么?协调器会分析你的数据,在成本-质量前沿上挑选候选模型,生成并验证评测工具,测量基线,针对失败案例迭代修改提示词,然后撰写报告。
为什么有三个独立的模型角色?智能体模型负责驱动聊天,目标模型是被测试的对象,评判模型则为 LLM-as-a-judge 指标打分。这是设计中的三种不同角色。
提示词优化如何决定修改什么?它会读取通过的示例、失败案例摘要和回归差异,然后做出最小的修改,以解决占主导地位的失败模式。
作者
awslabs
大小
3.1 MB
星标
552
更新
提交Nov 05
时长
22天
提交 · 90 天