reef README

The project's README from github.com/Human-Agent-Society/reef. As one markdown file: README.md.

Reef

Continual learning infra for self-improving agents

CI PyPI package: reef-infra Python License

English | 中文

Reef is the first open-source infrastructure for continual self-improving agents. It connects agent inference, feedback, learning, and versioned delivery. Use it to train model weights with Slime and SGLang, or improve an agent's harness, including its prompts, rules, and skills.

Get started | Roadmap | Launch post | Join Discord | Join WeChat Group

When to use Reef

Use Reef when you want your agent to keep improving simply by learning from how you interact with your agent.

Your goal Learning path What you need
Keep getting stronger model designed for you Model weight training A trainable model, a supported GPU stack, and feedback your recipe can use
Get your harness to self-improve Harness optimization A model endpoint, representative tasks, and an evaluator; no local training GPUs
Scientific discoveries Test-time training An execution environment, a correctness checker, and a measurable objective

How Reef fits your stack

Ability Inference engine (vLLM, SGLang, …) RL training framework (Slime, veRL, AReaL, …) Reef
Serves live traffic ✅ ❌ ✅
Trains weights ❌ ✅ ✅
Version management ❌ ❌ ✅
Stays live through updates ❌ ❌ ✅
Evolves beyond weights (skills, harness) ❌ ❌ ✅

How it works

Reef serves requests, records feedback, produces updates, and commits accepted updates to a version history.

Reef processes each learning cycle in four steps. The table also shows which modules implement each step.

Step What happens Where it lives
1 · Serve Serve agent requests and record interactions. service/ — agent requests and interaction records
runtime/ — inference and artifact updates
2 · Observe Match feedback to recorded interactions. records.py — stored interactions and feedback
train/processors/ — feedback matching and eligibility
3 · Grow Produce an update from eligible records. recipe/ — recipe integration
train/ — batches and update jobs
4 · Commit Apply the configured selection policy and publish accepted updates. train/evaluation/ — candidate evaluation
artifact/ — version history
surface/ — artifact delivery

Installation

💡 Note

Reef's artifact and checkpoint functionality requires the git-lfs system package. Reef initializes Git LFS locally for its artifact repositories.

We recommend uv for managing packages, and the commands below use it.

From PyPI

uv venv && source .venv/bin/activate
uv pip install reef-infra
python3 -c "import reef; print(reef.__version__)"

From source

git lfs install
git clone https://github.com/Human-Agent-Society/reef.git
cd reef
uv venv && source .venv/bin/activate
uv pip install -e .
python3 -c "import reef; print(reef.__version__)"

Use the source checkout for development and for the training examples below.

Using Reef

Reef supports two learning surfaces: model weights and agent harnesses. The deployment's recipe determines which surface its scenarios update.

Weight-training deployment

Start the deployment

The following example starts the SAO (arXiv:2607.07508) example deployment. Run it from a Reef checkout in an environment that satisfies the GPU requirements in Evolve your model.

uv pip install -e ".[slime]" && uv pip install --no-deps --group runtime

export MODEL_PATH="Qwen/Qwen2.5-1.5B-Instruct"
export REEF_TOKEN="reef-local"

reef serve -c recipes/sao/examples/sao/serve.yaml \
  --reef.model_path "$MODEL_PATH" \
  --reef.port "8900"

curl -f http://127.0.0.1:8900/healthz          # ready to serve

Send an inference request and report feedback

Send inference requests through Reef and report a score for each response. The SAO recipe uses each eligible scored rollout to run a training step.

Reef's inference endpoint is OpenAI- and Anthropic-compatible: /v1/chat/completions and /v1/messages take the provider's own request body. A request includes the x-reef-scenario header; a new name creates a scenario using the deployment's configured recipe. Requests do not select recipes.

The response body uses the provider's OpenAI-compatible format. Reef adds the x-reef-agent-record-id response header. Its value is the receipt that a later report uses to identify this interaction. A report can contain a numeric score, textual or structured feedback, and the receipts it evaluates. This example reports both a score and a short explanation.

import os
import httpx

reef = httpx.Client(
    base_url="http://127.0.0.1:8900",
    headers={"Authorization": f"Bearer {os.environ['REEF_TOKEN']}", "x-reef-scenario": "hello-reef"},
    timeout=300,
)

# Send a provider-compatible inference request
response = reef.post(
    "/v1/chat/completions",
    json={
        "model": os.environ["MODEL_PATH"],
        "messages": [{"role": "user", "content": "Return exactly: reef is ready"}],
    },
)

response.raise_for_status()
receipt = response.headers["x-reef-agent-record-id"]
answer = response.json()["choices"][0]["message"]["content"]

# Sending report about the inference
matched = answer.strip() == "reef is ready"

reef.post(
    "/reef/report",
    json={"score": float(matched), "feedback": "matched" if matched else "wrong answer", "references": [receipt]},
).raise_for_status()

feedback carries the richer signal, plain text or a structured object, for recipes that read more than a scalar. The endpoint will validate the report schema (reef/core/reports/).

Watch it learn and grow

Once the recipe has enough feedback, it runs a training step and synchronizes the updated weights to the serving runtime. Later inference requests use the current version without restarting Reef.

Harness-evolving deployment

Improve harness skills using a model API instead of GPUs.

Pass the model at startup using REEF_UPSTREAM_MODEL; no YAML edit is needed. deployment.yaml uses this variable for both serving and evaluation. From your Reef checkout and activated Python environment, replace the model ID and API key below with your provider's values:

export REEF_UPSTREAM_URL="https://api.openai.com"  # No /v1 suffix
export REEF_UPSTREAM_MODEL="REPLACE_WITH_YOUR_PROVIDER_MODEL_ID"
export REEF_UPSTREAM_API_KEY="your-openai-api-key"
reef serve -c tutorials/evolve-your-harness/configs/deployment.yaml

Use the exact model ID accepted by your provider, not the placeholder above. An unset or empty REEF_UPSTREAM_MODEL is reported at startup.

For another provider, use its base URL, model name, and API key. This config deploys Reef on 8901 with reef-local as its access token.

In another terminal, install the harness and run a task:

export REEF_TOKEN="reef-local"   # the script writes it into the installed harness's
                                 # model binding; keep it exported for `report`
curl -fsS -H "Authorization: Bearer $REEF_TOKEN" \
  'http://localhost:8901/reef/harness/install?adapter=pi' | bash
reef-pi -p "fix the failing test in auth.py"

# After running your tests, report the actual result:
reef-pi report --score 0 --feedback "missed the empty-token case"

If you already installed the harness with your-model-name, set REEF_UPSTREAM_MODEL, stop and restart reef serve with the command above, then rerun the harness install command before retrying reef-pi. Installation writes the model ID into the local harness configuration.

Failed reports trigger a candidate skill update. Reef evaluates it against the current harness on the tutorial's three coding tasks and publishes it only if it wins. See the tutorial to customize the tasks and evaluation.

To ask for a harness change in plain words and see the whole path from the ask to the install, run the harness requests tutorial.

Recipes and examples

Choose a recipe based on your workload's feedback and the artifact you want to update. The implementations live in this repository's recipes/ cookbook, are selected by dotted class reference, and do not ship in the Reef wheel.

Workload Recipe guide Updated artifact Examples and results
A stream of tasks scored by tests or a verifier SAO Model weights Example · Results
Agent traffic with useful next-state signals and no explicit reports OpenClaw-RL Model weights Example
Repeated, scored attempts at one problem TTT-Discover Model weights Example · Results
Parallel coding agents on one task, attempts scored by a grader (CORAL) CORAL TTT Model weights Example
Scored code search with a trainable guidance model and a frozen executor Guidance-TTT / TTTD Guidance-model weights Example · Results
Agent feedback used to evolve its skill pool SkillClaw Harness skills; no training GPUs Example
Scores and transcripts used to improve prompts and instructions GEPA Harness; fixed model weights Example and results

For a small walkthrough of feedback, candidate edits, and publication, start with the coding harness tutorial. Each result page documents its task, evaluation setup, measurements, and limitations.

Architecture

Reef architecture: harness requests flow through a scenario to inference. Receipt-linked feedback feeds records and recipe training; artifact evaluation selects updates for versioned publication. Rejected candidates leave the current release serving.

Learn more

The documentation is organized in the following order:

Community & Contributing

Working on continual self-improving agent?

If Reef looks useful to you, please give it a ⭐ — it helps the community to discover and contribute to the project.

The Team

Reef brings together people exploring how agents can learn from experience and improve over time. The people below help turn that idea into working infrastructure.

This list is non-exhaustive, with team members listed alphabetically by last name:

Wenhao Chai, Shuangrui Ding, Hao He, Haoze He, Chonghe Jiang, Nan Jiang, Xuan Jiang, Xiaochen Li, Paul Liang, Bo Liu, Boyuan Long, Qiuyang Mang, Zhenting Qi, Ao Qu, Mingruo Qu, Zhaokai Wang, Xuezhi Yan, Hanfei Yu, Haofei Yu, Simon Yu, Han Zheng, Kaichen Zhou, Zijian Zhou, Jiacheng Zhu, Dingyi Zhuang.

Star History

Reef Star History Chart

Acknowledgements

We are particularly grateful to these projects which power important parts of Reef:

reef (Human-Agent-Society/reef)

vllm
Searching Database…

7.6K ·GitHub
Reef
What is this?Reef is open-source infrastructure for continually self-improving agents. It wires together agent inference, feedback collection, learning, and versioned delivery so an agent keeps getting better from how you interact with it.
Is Reef a training framework, a serving layer, or both?Both: Reef serves live agent traffic like an inference engine while also managing training and version delivery, which plain engines or RL frameworks alone do not do.
How do I install Reef and get a deployment running?Install it from PyPI or clone the repo for a source checkout (needed for the training examples), using uv to manage packages, then start a deployment with 'reef serve' against a deployment.yaml config.
What do I need installed before running Reef?You need the git-lfs system package for artifact and checkpoint storage, plus either a trainable model with a supported GPU stack (for weight training) or a model API endpoint and key (for harness evolution).
Do I need GPUs to use Reef?Only for the weight-training path, which must run in an environment meeting the GPU requirements for Slime/SGLang training; harness-evolving deployments instead just need a model API and no local training GPUs.
Do I run Reef myself, or is it a hosted service?Reef is self-hosted: you run 'reef serve' on your own infrastructure, exposing OpenAI- and Anthropic-compatible endpoints ('/v1/chat/completions' and '/v1/messages') for your agent traffic.
What recipes come with Reef out of the box?The repo ships a recipes/ cookbook including SAO, OpenClaw-RL, TTT-Discover, CORAL TTT, Guidance-TTT/TTTD, SkillClaw, and GEPA, covering both model-weight training and harness skill evolution.
How is this different from just running vLLM or SGLang?Inference engines like vLLM or SGLang only serve traffic; Reef adds feedback matching, training triggers, version management, and staying live through updates on top of serving.
Do I need to restart Reef after it learns something new?No. Once a recipe trains new weights or a candidate harness skill wins its evaluation against the current one, Reef publishes it and subsequent requests automatically use the new version.
Can I improve my agent's prompts without training a model?Yes. A harness-evolving deployment improves prompts, rules, and skills using a model API endpoint, with no local training GPUs required.
What are similar projects?OpenClaw-RL, verl, rrsi, Agent-Reflex, ms-swift, agent-lightning, ApexAgents-SkyRL-Recipe, DeepAnalyze
Which topics is it in?agent-infrastructure, continual-learning, self-improving-agents, llm-training, sglang, model-training, agent-harness, training, vllm, harness, agent-framework, reinforcement-learning
Creator
Human-Agent-Society
Size
38.3 MB
Stars
7.6K
Updated
CommitOct 19Releasev0.1.3Oct 14
Age
1mo
Commits · 90 Days