shrt README

项目的 README,来自 github.com/N4darae/shrt。合并为一个 markdown 文件:README.md。

shrt

shrt tests a backend through chains of API calls. A chain is a YAML file: log in, create an item, add stock, read it back. You run the chains, a person approves a passing run as the baseline, and every later release is replayed against that baseline. When a field changes, shrt names the step, the field, and the write it suspects.

It was built to be driven by a coding agent (shrt init installs a Claude Code skill and its docs), but every command works by hand, and the demo below is all by hand.

shrt speaks the Connect protocol: each call is JSON over HTTP POST. A server built with connect-go answers it, since connect-go serves Connect and gRPC side by side. A server that speaks only gRPC (binary frames over HTTP/2, as plain grpc-go does) does not. Unary and server-streaming rpcs are supported; a streaming step reads the first message.

Install

You need Go 1.26 or later, and buf on your PATH: shrt compiles the backend's .proto files with buf build to learn every rpc and message.

go install github.com/N4darae/shrt/cmd/shrt@latest
shrt version

Or build from a checkout with go build -o shrt ./cmd/shrt and put the binary on your PATH.

A ten-minute demo

The demo runs shrt against examples/inventory, a small connect-go service with four rpcs: Login, CreateItem, GetItem and AddStock. Started with -release 2, it serves GetItem from a stale cache, and that is the regression the demo catches.

Copy the example out of the checkout first. shrt init writes .shrt/, .claude/ and a .gitignore into the directory it runs in.

git clone https://github.com/N4darae/shrt
cp -r shrt/examples/inventory /tmp/inventory-demo
cd /tmp/inventory-demo

1. Start release 1

go build -o inventory . && ./inventory &

It listens on 127.0.0.1:8080, which is also where shrt init looks by default. If that port is taken, start it with -addr 127.0.0.1:9090 and pass -base-url http://127.0.0.1:9090 to shrt init.

2. Set up shrt

export API_USER=demo API_PASSWORD=demo-password
shrt init
shrt doctor

init builds the descriptor from the protos, guesses which rpc is the login, and logs in once to see how responses report success. Two of its lines say what it found:

write .shrt/config.yaml auth: inventory.v1.AuthService/Login, GUESSED from the descriptor: check it and the credential variables it names
write .shrt/config.yaml conventions: envelope_path status.code, envelope_ok OK (read from one Login answer)

Credentials stay in the environment. The config refers to them as ${env.API_USER} and ${env.API_PASSWORD}, and passwords and tokens are redacted from every run record.

3. Add the contract

A contract says what each rpc should do: which fields it requires, where each value comes from, how it refuses, and what a write does to numbers. shrt contract init -all writes a skeleton full of TODOs, which you (or an agent) fill in from the service's code. The example ships a filled-in one:

shrt contract init -all
cp contracts/inventory.yaml .shrt/contracts/
shrt contract lint

The line that matters most for this demo is in AddStock: effects: {qty_on_hand: {increase: qty}}. It tells the planner that stock read after an AddStock should have grown by qty.

4. Let the contract write the chains

shrt contract plan -all -write

This writes one chain per rpc, four chains and 53 steps in all. Each one calls its rpc the normal way and then probes it: the boundary values in the contract, a missing token and a bad one, an id nobody created, a duplicate sku. Around the first AddStock call the planner reads the item once before and once after, and the read after asserts the stock the effects line works out.

5. Run everything and approve the baseline

shrt gate
shrt confirm -all -note "release 1 of the inventory demo"
shrt confirm -all -approve -by [email protected]

gate sends every chain once, and all four pass. confirm -all proposes each passing run as a chain's safe spot and prints a table of what was run and which refusals each chain asserts. Only a person approves: an agent driving shrt is told to show that table and wait for a yes. Once approved, each response field becomes part of the baseline, including fields no assertion checks.

6. Ship a regression

kill %1
./inventory -release 2 &
shrt gate
FAIL       inventory-addstock    regression: get_item_after_add_stock (ItemService/GetItem) item.qty_on_hand want=5 got=0; unclear: write add_stock (ItemService/AddStock) or the read
PASS       inventory-createitem
PASS       inventory-getitem
PASS       inventory-login
...
shrt gate: FAIL: 1 of 4 chain(s) failed; details: shrt verify <chain>, or shrt run <chain> -keep-going

Restarting the service threw away the token shrt had cached. The gate notices the refusal and may hold a fresh token for up to 30 seconds to tell a restart from sessions that end too early; that is the pause you see. The gate exits 1, so in CI it fails the build.

shrt verify inventory-addstock shows the evidence. AddStock answered 5, but the read after it still says 0, the stock the read before it saw:

inventory-addstock: DRIFT (regression), 1 step(s) changed vs safe spot ...; first: get_item_after_add_stock (ItemService/GetItem) item.qty_on_hand want=5 got=0; unclear: write add_stock (ItemService/AddStock) or the read: answered qty_on_hand=5, but GetItem read 0
  add_stock sent {"id_item":"item-11","qty":"5"}
  [get_item_after_add_stock] changed    item.qty_on_hand want=5 got=0

Stop the service with kill %1 when you are done. In a real repository you commit .shrt/config.yaml, the contracts, the chains and the safe spots, and run .shrt/ci-gate.sh (written by init) in CI against each new release.

Keeping it away from production

Chains create records and send deliberately bad requests, so pointing one at production would write test data there. shrt only sends to this machine (localhost, 127.0.0.1, ::1) unless the host is listed in the config:

target:
    base_url: https://staging.example.com
    allow_hosts: [staging.example.com]

With a host missing from that list, run, verify, gate and init's login all refuse before the first call and print the line to add. run -dry-run still works, because it sends nothing. The same check applies to target.host_override. shrt cannot tell staging from production by itself, so list only backends whose data you are willing to have changed.

Using it with an agent

shrt init installs .claude/skills/shrt/SKILL.md, a subagent that writes contracts, and the agent docs under .shrt/docs/. The sources of those docs are in docs/agent/: the command reference, every config and chain key, the procedures, and a symptom-to-fix list. The design and its limits are in docs/OVERVIEW.md.

Every command prints its flags and exit codes with -h.

License

MIT. See LICENSE.

shrt (N4darae/shrt)

正在搜索数据库…

2.9K ·GitHub
这是什么?shrt 是一个为 AI 编程代理打造的、基于契约的 gRPC 回归测试工具,基线需经人工批准。它会为一个 API 的有效调用链拍下快照,使代理可以直接查阅,而无需从头摸索。
这是命令行工具还是库?它是一个命令行工具:一个 `shrt` 可执行文件,可通过 `init`、`run`、`verify` 和 `gate` 等子命令运行;另外还有一个技能文件,用于将 AI 代理引导至其文档。
怎样才能在仓库里跑起来?运行 `shrt init`,即可生成 `.shrt/` 目录、构建 proto 描述符,并安装技能/子代理以及 `.shrt/ci-gate.sh`。每次修改 proto 后都需要重新构建描述符;每次修改 shrt 本身后,则需用 `go build -o shrt ./cmd/shrt` 重新构建二进制文件。
运行测试前需要先做什么设置吗?需要——在任何运行之前,你必须先导出 `auth.body` 所读取的环境变量(变量名来自 `.shrt/config.yaml`);没有这些变量,`shrt run` 会拒绝发送流量。
它也支持 REST/OpenAPI 接口吗?目前还不支持——暂时只支持 gRPC 服务,OpenAPI 支持计划作为未来的新增功能。
这是给人工测试人员用的,还是给 AI 编程代理用的?它主要面向 AI 编程代理而设计(附带位于 .claude/skills/shrt/SKILL.md 的 Claude 技能文件),用于组合和验证 RPC 调用链;但在任何通过的运行成为可信基线之前,必须由人工批准。
那四条不可妥协的规则是什么?只有在用户明确表示同意后才批准基线;绝不调换、跳过或并行执行步骤,因为顺序本身就是契约的一部分;绝不凭记忆手写请求体(字段必须始终从描述符中获取);也绝不让某个步骤只断言“它没有崩溃”。
`shrt gate` 实际检查什么?这是 CI 闸门:它会重放每一条调用链(若有安全点则通过 verify,否则通过 run),与已批准的基线进行比对,并报告 PASS、KEPT RED、FAIL(回归或 pin-break)或 FINDING(间歇性失败);一旦出现回归,便会导致构建失败。
什么是“契约”?为什么要用它,而不是手写调用链?契约描述了预期行为(包括写操作对数值会产生什么影响),`shrt contract plan` 会据此自动组合调用链,覆盖手写调用链通常会遗漏的边界情况、缺失令牌的情况以及回读校验。
基线是如何获得批准的?调用链通过后,`shrt confirm -note "..."` 会将其提议为安全点,并附上一份通俗易懂的摘要;只有在用户回复“是”之后,`shrt confirm -approve -by` 才会真正写入已批准的基线。
作者
N4darae
大小
42.7 MB
星标
2.9K
更新
提交Oct 26
时长
2个月
提交 · 90 天