项目的 README,来自 github.com/awslabs/ai-eval-studio。合并为一个 markdown 文件:README.md。
An end-to-end LLM evaluation platform. Upload a dataset, describe your goal in chat, and the system runs evaluations across models, optimizes the prompt iteratively against real failures, and hands you back a standalone Python eval harness you can run in CI.
Every model call — the agent driving the conversation, the models under test, and the LLM judge — runs on Amazon Bedrock.
This project is a prototype for demonstration, not for production use. Before using it with real data or real users, do your own independent assessment against your security and quality standards.
SECURITY.mdlists what the default deployment does and does not do, and the steps to take it to production.
You talk to a coordinator agent, which walks the project through seven phases:
| Phase | What happens |
|---|---|
| 1. Data analysis | Profiles your uploaded CSV/Excel and proposes field mappings |
| 2. Configuration | Picks candidate models on a cost-quality frontier and chooses metrics |
| 3. Eval harness | Generates a standalone DeepEval harness and validates it runs |
| 4. Baseline prompt | Drafts the prompt to optimize from |
| 5. Confirm & run | Measures a real baseline across the candidate pool |
| 6. Eval + optimize | Iterates prompt edits against failures until scores converge |
| 7. Report | Produces a cost/quality report and a recommended model + prompt |
The generated harness is the durable artifact: a self-contained Python project that
runs without the platform. See
backend/app/templates/harness/README.md.
The platform runs in two modes from one codebase, selected by EVAL_STUDIO_MODE:
dev — everything in one FastAPI process on your machine, projects on local disk.cloud — a Fargate coordinator behind CloudFront; agent turns run in a per-user
Bedrock AgentCore microVM, and each eval job runs as its own Fargate task.Click to view full size. Editable source:
architecture-overview.drawio (draw.io).
Components
/ws) carries
chat, job, and project events.Prerequisites — Python 3.11+, uv, Node.js 22+, git,
and AWS credentials.
Anthropic models need a one-time use-case form. Bedrock enables its serverless models automatically, but each account must submit Anthropic's first-time-use form before the first call to a Claude model; submitting it from an organization's management account covers its member accounts. The coordinator agent runs on Claude, so submit the form before the first chat. See Access Amazon Bedrock foundation models.
The local agent runs shell commands on your machine without confirmation. The coordinator writes and executes an evaluation harness, and its
Bashtool is not sandboxed or path-restricted — it runs with your user's privileges. Run it on a machine where that is acceptable.
(cd backend && uv sync)
(cd frontend && npm install)
./scripts/start.sh # starts backend on :8100 and frontend on :3100
./scripts/stop.sh
Then open http://localhost:3100. start.sh exports CLAUDE_CODE_USE_BEDROCK=1, which
the Agent SDK requires.
Both processes log to the terminal and append to .dev-logs/backend.log and
.dev-logs/frontend.log (gitignored), so a failure stays readable by someone who
wasn't at the keyboard when it happened.
The cloud deployment is AWS CDK (Python) across nine stacks — networking, storage, auth, IAM, observability, compute, frontend, AgentCore, and an AppRegistry application.
infra/README.md — CDK prerequisites, synth, and the
deploy/iterate workflow.docs/runbooks/deploy.md — first full cloud deploy,
start to finish.docs/runbooks/teardown.md — full teardown.The deployed entry point is the CloudFront URL; the ALB is internal and not reachable
from the internet. Sign-in is Cognito hosted UI (OIDC + PKCE). The default deployment is
a development deployment; SECURITY.md lists the steps to take it to
production.
Treat one deployment as a single trust domain. Everyone you onboard can reach everyone else's data. The platform is multi-user — Cognito sign-in, per-user projects — and every API route and agent tool call checks ownership before it answers, but an application-layer check is not a boundary the agent itself is confined by: the product generates and runs code, with a shell, by design.
Three specific gaps, named rather than summarised, because "app-layer isolation only" reads as vaguer than the truth:
Write, Edit and NotebookEdit are path-jailed to the
project directory, and tool calls are pinned to the project the conversation is about
— but Bash reads /mnt/projects directly and passes through neither. That is
deliberate: a static scan of a command line cannot contain .., $VAR or $(...),
so scanning it would assert a guarantee it does not deliver
(agent_runner.py). Uploaded datasets,
prompts, generated harnesses and reports all live here — the one layer with no IAM
surface.iam_stack.py).The second can be closed with AssumeRoleWithWebIdentity, carrying the tenant tag in
a claim inside the Cognito ID token so STS verifies it and the caller stops asserting it.
The first needs per-session filesystem storage instead of a shared mount; AgentCore
offers one and it is in preview. The first is what multi-tenant use actually depends
on, and it is not done.
Two related properties are deliberate rather than oversights:
(cd backend && uv run pytest) # backend suite
(cd frontend && npm test) # frontend suite (vitest)
(cd frontend && npm run build) # eslint + tsc + vite build
./scripts/lint.sh # ruff over backend/, infra/, scripts/
./scripts/test-infra.sh # CDK synth-level tests (see note)
The infra tests need a runner because they sit between two virtualenvs: they
import aws_cdk (installed in infra/.venv) and are collected by pytest
(installed in backend/.venv). test-infra.sh puts the former on PYTHONPATH
for the latter, so neither venv has to duplicate the other's dependencies.
Tests marked integration hit real external services and are skipped by default; run
them with uv run pytest -m integration.
├── frontend/ # React 19 + Vite + TypeScript
├── backend/
│ └── app/
│ ├── api/ # Route handlers (chat, eval, projects, upload, ws)
│ ├── agents/ # Subagent definitions
│ ├── prompts/ # Coordinator + subagent system prompts (markdown)
│ ├── models/ # Pydantic schemas
│ ├── platform/ # dev vs cloud adapters (storage, auth, compute)
│ ├── services/ # Core logic (agent runner, jobs, reports, MCP tools)
│ └── templates/ # Standalone eval harness template
├── infra/ # AWS CDK app (nine stacks)
├── scripts/ # start.sh, stop.sh, lint.sh, test-infra.sh
├── projects/ # User eval projects (gitignored)
└── docs/ # Runbooks, diagrams, reference
| Doc | What it covers |
|---|---|
SECURITY.md |
Reporting a vulnerability; path to production |
infra/README.md |
CDK layout, prerequisites, deploy/iterate loop |
docs/runbooks/deploy.md |
First cloud deploy, step by step |
docs/runbooks/teardown.md |
Full teardown |
docs/error-codes.md |
Stable ResourceError.code catalog |
docs/naming-conventions.md |
File and symbol naming rules |
docs/frontend-design-system.md |
Theming tokens and the no-hardcoded-color rule |
docs/diagrams/ |
Cloud architecture diagram |
This project is licensed under the Apache-2.0 License. See LICENSE,
NOTICE and THIRD-PARTY-LICENSES.
This project depends on the Claude Code CLI, which is proprietary to Anthropic PBC and not covered by this project's license. You must obtain and use it under Anthropic's Commercial Terms of Service.
See SECURITY.md for how to report a potential security issue. Please do
not open a public issue.
Known limitations of the deployed system are stated in Security scope above — read that before deploying this for more than one person.