Reef is the first open-source infrastructure for continual self-improving agents.
It connects agent inference, feedback, learning, and versioned delivery. Use it
to train model weights with Slime and SGLang, or improve an agent's harness, including its prompts, rules, and skills.
Use the source checkout for development and for the training examples below.
Using Reef
Reef supports two learning surfaces: model weights and agent harnesses.
The deployment's recipe determines which surface its scenarios update.
Weight-training deployment
Start the deployment
The following example starts the SAO (arXiv:2607.07508) example deployment. Run it
from a Reef checkout in an environment that satisfies the GPU requirements in
Evolve your model.
Send inference requests through Reef and report a score for each response. The
SAO recipe uses each eligible scored rollout to run a training step.
Reef's inference endpoint is OpenAI- and Anthropic-compatible: /v1/chat/completions
and /v1/messages take the provider's own request body. A request includes the
x-reef-scenario header; a new name creates a scenario using the deployment's
configured recipe. Requests do not select recipes.
The response body uses the provider's OpenAI-compatible format. Reef adds the
x-reef-agent-record-id response header. Its value is the receipt that a
later report uses to identify this interaction. A report can contain a numeric
score, textual or structured feedback, and the receipts it evaluates. This
example reports both a score and a short explanation.
feedback carries the richer signal, plain text or a structured object,
for recipes that read more than a scalar. The endpoint will validate the
report schema (reef/core/reports/).
Watch it learn and grow
Once the recipe has enough feedback, it runs a training step and synchronizes
the updated weights to the serving runtime. Later inference requests use the
current version without restarting Reef.
Harness-evolving deployment
Improve harness skills using a model API instead of GPUs.
Pass the model at startup using REEF_UPSTREAM_MODEL; no YAML edit is needed.
deployment.yaml uses this
variable for both serving and evaluation. From your Reef checkout and activated
Python environment, replace the model ID and API key below with your provider's values:
Use the exact model ID accepted by your provider, not the placeholder above.
An unset or empty REEF_UPSTREAM_MODEL is reported at startup.
For another provider, use its base URL, model name, and API key. This config deploys Reef on 8901 with reef-local as its access token.
In another terminal, install the harness and run a task:
export REEF_TOKEN="reef-local" # the script writes it into the installed harness's
# model binding; keep it exported for `report`
curl -fsS -H "Authorization: Bearer $REEF_TOKEN" \
'http://localhost:8901/reef/harness/install?adapter=pi' | bash
reef-pi -p "fix the failing test in auth.py"
# After running your tests, report the actual result:
reef-pi report --score 0 --feedback "missed the empty-token case"
If you already installed the harness with your-model-name, set REEF_UPSTREAM_MODEL,
stop and restart reef serve with the command above, then rerun the harness install
command before retrying reef-pi. Installation writes the model ID into the local
harness configuration.
Failed reports trigger a candidate skill update. Reef evaluates it against the
current harness on the tutorial's three coding tasks and publishes it only if
it wins. See the tutorial to customize the
tasks and evaluation.
To ask for a harness change in plain words and see the whole path from the ask to the install, run the harness requests tutorial.
Recipes and examples
Choose a recipe based on your workload's feedback and the artifact you want to
update. The implementations live in this repository's recipes/ cookbook,
are selected by dotted class reference, and do not ship in the Reef wheel.
For a small walkthrough of feedback, candidate edits, and publication, start with
the coding harness tutorial. Each result
page documents its task, evaluation setup, measurements, and limitations.
Architecture
Learn more
The documentation is organized in the following order:
Quickstart: install Reef, connect a client, and inspect the version history
Report suspected vulnerabilities privately by following the security policy.
If Reef looks useful to you, please give it a ⭐ — it helps the community to discover and contribute to the project.
The Team
Reef brings together people exploring how agents can learn from experience and
improve over time. The people below help turn that idea into working infrastructure.
This list is non-exhaustive, with team members listed alphabetically by last name:
Reef 是一套开源基础设施，用于让智能体持续自我改进。它把智能体推理、反馈收集、学习和带版本的交付串联起来，让智能体随着你与它的互动不断变好。