简体中文 | English
An AI harness reference implementation for on-demand delivery operations. It combines deterministic workflows, mock business tools, lightweight knowledge retrieval, an OpenAI-compatible model endpoint, evaluation scaffolding, and human-review guardrails.
[!IMPORTANT]
This repository is an educational MVP, not a production-ready delivery or compensation system. Business tools use synthetic data, most state is held in memory, and model output is advisory. Do not expose the application to the public internet, submit real personal/order data, or execute compensation decisions without authentication, policy enforcement, and human approval.
What is implemented
- Two synchronous workflows: abnormal-order analysis and compensation suggestion.
- Four locally registered tools for orders, ETA, station capacity, and compensation rules. Every tool derives its result from its arguments.
- Synthetic orders that vary by order ID across six named scenarios, including one delivered on time.
- A deterministic timeline that splits an order into dispatch wait, to-shop, merchant prep, and on-road legs, reported alongside the model's answer as the baseline it has to beat.
- Compensation amounts decided by the rule engine, never by the model (ADR-0002).
- OpenAI-compatible chat-completions client behind an interface, so workflows are testable without a model.
- In-memory document ingestion, text chunking, lexical retrieval scored by query-term overlap, and a rule and case base seeded at startup.
- In-memory evaluation runs whose scorers report "not measured" rather than a perfect score when a case declares no expectation.
- Request trace IDs, per-step durations, a bounded trace store, per-scenario metrics, feedback storage, and advisory output checks.
- Completed workflows with an unavailable optional dependency are reported as
DEGRADED; the deterministic result remains available and the affected step is recorded as failed.
- A local Ollama profile, 112 tests, Maven Wrapper, and GitHub Actions CI.
The following are intentionally not claimed as implemented: autonomous agent planning, production tool integrations, durable persistence, vector embeddings/Milvus RAG, authentication, rate limiting, distributed tracing, or automatic compensation execution. See Known limitations.
Architecture
flowchart LR
Client["REST client"] --> Api["harness-api\nHTTP, validation, tracing"]
Api --> Agent["core: agent\nfixed workflows"]
Agent --> Tools["core: tool\nsynthetic business tools"]
Agent --> Knowledge["core: knowledge\nseeded rules, lexical retrieval"]
Agent --> LLM["core: llm\nOpenAI-compatible client"]
Agent --> Observe["core: observe\ntrace and metrics"]
LLM --> Model["Ollama or compatible endpoint"]
Api --> Eval["core: eval"]
Eval --> Agent
All workflow steps run synchronously in the gateway process. Apart from the configured model endpoint, the default application has no required external services.
Modules
| Module |
Responsibility |
harness-common |
DTOs, exceptions, JSON/text helpers, and trace context. No internal dependencies. |
harness-core |
Everything that decides something: agent (workflows, timeline, guardrails, formatting), tool (synthetic business tools), llm (model routing and transport), knowledge (seeded rules and cases, lexical retrieval), eval (cases, runs, scorers), observe (traces, metrics, feedback). |
harness-api |
Executable Spring Boot application: controllers, validation, exception mapping, request tracing. |
llm-inference |
Pinned Ollama container definition and smoke-test script. |
Java packages are unchanged — com.delivery.harness.{agent,tool,llm,knowledge,eval,observe} still exist and still mean the same thing. Only the Maven build boundary moved; see ADR-0004.
Prerequisites
- JDK 17 or newer
- Maven 3.6.3 or newer, or the included Maven Wrapper
- Docker with Compose support, only when running Ollama in a container
- Enough memory for the model you choose;
qwen2.5:7b is the default example
Quick start
1. Start a local model
If Ollama already runs on your machine, skip the first command.
docker compose up -d ollama
docker compose exec ollama ollama pull qwen2.5:7b
The Compose port binds to 127.0.0.1 only. Model weights are downloaded separately and are not part of this repository or its MIT license.
2. Build and test
./mvnw clean verify
3. Run the API
./mvnw -pl harness-api -am package
java -jar harness-api/target/harness-api.jar
The API listens on http://localhost:8080 by default.
Release artifacts
Tagged releases publish the runnable harness-api fat JAR and a SHA-256
checksum as GitHub Release assets. This is a 0.x release: the HTTP surface
and response fields may change without a major-version bump. Pin a tag if you
depend on the response shape.
4. Call a workflow
curl --fail-with-body \
--request POST \
--header 'Content-Type: application/json' \
--data '{"order_id":"DEMO-001"}' \
http://localhost:8080/api/v1/analyze/abnormal-order
Compensation suggestion:
curl --fail-with-body \
--request POST \
--header 'Content-Type: application/json' \
--data '{"order_id":"DEMO-002","complaint_type":"OVERTIME"}' \
http://localhost:8080/api/v1/analyze/compensation
For DAMAGED complaints, pass "damage_level":"FULL" or
"damage_level":"PARTIAL". Without evidence, the policy returns zero and
requires manual review; the rule engine, not the model, remains authoritative.
Every response uses an envelope similar to:
{
"code": 0,
"message": "success",
"data": {
"executionId": "...",
"traceId": "...",
"status": "SUCCESS",
"steps": []
},
"traceId": "..."
}
The same trace ID is returned in the X-Trace-Id response header.
API surface
| Method |
Path |
Purpose |
POST |
/api/v1/analyze/abnormal-order |
Run abnormal-order analysis |
POST |
/api/v1/analyze/compensation |
Generate a compensation suggestion |
POST |
/api/v1/knowledge/ingest |
Ingest and chunk an in-memory document |
POST |
/api/v1/knowledge/search |
Search in-memory rules, documents, and cases |
POST |
/api/v1/eval/cases |
Add an in-memory evaluation case |
GET |
/api/v1/eval/cases |
List evaluation cases |
POST |
/api/v1/eval/run |
Run selected cases synchronously |
GET |
/api/v1/eval/run/{runId} |
Read an evaluation run |
GET |
/api/v1/eval/run/{runId}/results |
Read evaluation results |
GET |
/api/v1/observe/trace/{traceId} |
Read a recorded trace, when populated |
GET |
/api/v1/observe/traces |
List recorded traces, when populated |
GET |
/api/v1/observe/metrics |
Read in-memory metrics, when populated |
POST |
/api/v1/observe/feedback |
Store feedback in memory |
GET |
/api/v1/observe/feedback |
List in-memory feedback |
Configuration
Copy .env.example if you want a local reference. Spring reads the following environment variables directly:
| Variable |
Default |
Description |
SERVER_ADDRESS |
127.0.0.1 |
Bind address; set 0.0.0.0 only for an intentionally isolated remote/container deployment |
SERVER_PORT |
8080 |
API port |
HARNESS_LLM_ENDPOINT |
http://localhost:11434 |
OpenAI-compatible base URL |
HARNESS_LLM_MODEL |
qwen2.5:7b |
Default model name |
HARNESS_LLM_API_KEY |
empty |
Optional bearer token; never commit a real value |
HARNESS_LLM_TIMEOUT_SECONDS |
120 |
Model read timeout |
HARNESS_MAX_COMPENSATION_AMOUNT |
50.0 |
Advisory maximum amount check |
Additional Spring properties, settable in application.yml or as --property=value:
| Property |
Default |
Description |
harness.compensation.approval-threshold |
10.0 |
Payouts at or above this value require a human approver |
harness.knowledge.seed.enabled |
true |
Load the synthetic rule and case base at startup |
harness.eval.seed.enabled |
true |
Load the starter evaluation cases at startup |
harness.observe.max-traces |
500 |
Size of the in-memory trace ring |
harness.observe.max-feedback |
1000 |
Maximum in-memory feedback records |
harness.observe.max-tool-invocations |
2000 |
Maximum recent tool invocation records |
harness.eval.max-cases |
1000 |
Maximum in-memory evaluation cases |
harness.eval.max-runs |
200 |
Maximum retained in-memory evaluation runs |
harness.knowledge.max-documents |
1000 |
Maximum in-memory knowledge documents |
harness.knowledge.max-chunks |
5000 |
Maximum in-memory knowledge chunks |
Run the model smoke test after installing the configured model:
HARNESS_LLM_MODEL=qwen2.5:7b ./llm-inference/smoke-test.sh
Testing
./mvnw clean verify
The suite contains 112 tests. Both workflows are covered end to end against a stub model transport, so no test requires Ollama or a Docker daemon. The load-bearing test is buildsDifferentEvidenceForDifferentOrders: it asserts that two different orders produce two different prompts, which is the property every other measurement depends on. CI runs the same Maven verification on every push and pull request.
Security and data handling
- There is no authentication, authorization, tenant isolation, or rate limiting.
- Tool data and addresses are synthetic examples; do not replace them with real personal data in a public deployment.
- Ingested documents can influence prompts. Treat knowledge ingestion as a prompt-injection trust boundary.
- The advisory checker detects a small set of phrases and amount conditions; it is not a policy engine.
- Most in-memory stores are demonstration components and are cleared on restart; traces and feedback have explicit capacity limits, while evaluation and knowledge stores still need lifecycle governance.
- Review SECURITY.md before reporting a vulnerability or deploying a derivative.
Known limitations
- Tool implementations return synthetic data rather than calling delivery systems. The data varies with its arguments, which makes the pipeline testable — it does not make it accurate.
- Retrieval is lexical term overlap, not semantic search. CJK text is split on whitespace and punctuation rather than segmented, so recall depends on the query and the rule sharing a phrase (ADR-0003).
- Rules, documents, cases, evaluations, feedback, metrics, and traces are not durable and are lost on restart.
- The evaluation scorers are lexical and are calibrated to catch regressions, not to certify quality. Six synthetic cases cannot establish accuracy; a real claim needs a labelled set drawn from production traffic.
modelVersion and promptVersion on an evaluation run are recorded metadata labels; they do not yet select a model or mutate the workflow prompt. Pin those inputs through deployment configuration when comparing runs.needs_human_review and approval_required are derived from guardrail outcome, parse success, model confidence and baseline agreement. This is reporting, not authority: nothing here executes a payment.- Guardrails report pass/fail in workflow steps and the final output, but do not replace human review.
- There is no persistence layer. A prior revision shipped an unexecuted PostgreSQL schema; it was removed rather than left to imply otherwise.
Roadmap
- Add authenticated, authorized API access and request quotas.
- Replace mock tools with versioned adapters and contract tests.
- Add durable repositories and migrations behind explicit profiles.
- Persist traces, metrics, feedback, and audit events with retention limits.
- Introduce typed workflow outputs, stronger policy enforcement, and redaction.
- Build a labelled evaluation set from real abnormal orders and measure top-1 attribution accuracy against the deterministic baseline. If the model does not beat that baseline, the model is not earning its place in the attribution path.
- Bind evaluation model/prompt versions to immutable runtime configuration so a run can be reproduced from its recorded metadata.
- Measure reviewer handling time before and after. Without it, the system cannot demonstrate that it saves anything.
- Embedding generation and vector retrieval are deliberately not on this roadmap at the current corpus size; see ADR-0003 for the conditions that would reopen it.
Contributing
Contributions are welcome. Read CONTRIBUTING.md, follow the Code of Conduct, and use SECURITY.md for security-sensitive reports.
License
The source code is licensed under the MIT License. Downloaded model weights and third-party services retain their own licenses; see THIRD_PARTY_NOTICES.md.
An AI harness reference implementation for on-demand delivery operations that combines deterministic workflows, mock business tools, lightweight knowledge retrieval, an OpenAI-compatible model endpoint, evaluation scaffolding, and human-review guardrails.