reef README

The project's README from github.com/Human-Agent-Society/reef. As one markdown file: README.md.

Reef

Continual learning infra for self-improving agents

CI PyPI package: reef-infra Python License

English | 中文

Reef is the first open-source infrastructure for continual self-improving agents. It connects agent inference, feedback, learning, and versioned delivery. Use it to train model weights with Slime and SGLang, or improve an agent's harness, including its prompts, rules, and skills.

Get started | Roadmap | Launch post | Join Discord | Join WeChat Group

When to use Reef

Use Reef when you want your agent to keep improving simply by learning from how you interact with your agent.

Your goal Learning path What you need
Keep getting stronger model designed for you Model weight training A trainable model, a supported GPU stack, and feedback your recipe can use
Get your harness to self-improve Harness optimization A model endpoint, representative tasks, and an evaluator; no local training GPUs
Scientific discoveries Test-time training An execution environment, a correctness checker, and a measurable objective

How Reef fits your stack

Ability Inference engine (vLLM, SGLang, …) RL training framework (Slime, veRL, AReaL, …) Reef
Serves live traffic ✅ ❌ ✅
Trains weights ❌ ✅ ✅
Version management ❌ ❌ ✅
Stays live through updates ❌ ❌ ✅
Evolves beyond weights (skills, harness) ❌ ❌ ✅

How it works

Reef serves requests, records feedback, produces updates, and commits accepted updates to a version history.

Reef processes each learning cycle in four steps. The table also shows which modules implement each step.

Step What happens Where it lives
1 · Serve Serve agent requests and record interactions. service/ — agent requests and interaction records
runtime/ — inference and artifact updates
2 · Observe Match feedback to recorded interactions. records.py — stored interactions and feedback
train/processors/ — feedback matching and eligibility
3 · Grow Produce an update from eligible records. recipe/ — recipe integration
train/ — batches and update jobs
4 · Commit Apply the configured selection policy and publish accepted updates. train/evaluation/ — candidate evaluation
artifact/ — version history
surface/ — artifact delivery

Installation

💡 Note

Reef's artifact and checkpoint functionality requires the git-lfs system package. Reef initializes Git LFS locally for its artifact repositories.

We recommend uv for managing packages, and the commands below use it.

From PyPI

uv venv && source .venv/bin/activate
uv pip install reef-infra
python3 -c "import reef; print(reef.__version__)"

From source

git lfs install
git clone https://github.com/Human-Agent-Society/reef.git
cd reef
uv venv && source .venv/bin/activate
uv pip install -e .
python3 -c "import reef; print(reef.__version__)"

Use the source checkout for development and for the training examples below.

Using Reef

Reef supports two learning surfaces: model weights and agent harnesses. The deployment's recipe determines which surface its scenarios update.

Weight-training deployment

Start the deployment

The following example starts the SAO (arXiv:2607.07508) example deployment. Run it from a Reef checkout in an environment that satisfies the GPU requirements in Evolve your model.

uv pip install -e ".[slime]" && uv pip install --no-deps --group runtime

export MODEL_PATH="Qwen/Qwen2.5-1.5B-Instruct"
export REEF_TOKEN="reef-local"

reef serve -c recipes/sao/examples/sao/serve.yaml \
  --reef.model_path "$MODEL_PATH" \
  --reef.port "8900"

curl -f http://127.0.0.1:8900/healthz          # ready to serve

Send an inference request and report feedback

Send inference requests through Reef and report a score for each response. The SAO recipe uses each eligible scored rollout to run a training step.

Reef's inference endpoint is OpenAI- and Anthropic-compatible: /v1/chat/completions and /v1/messages take the provider's own request body. A request includes the x-reef-scenario header; a new name creates a scenario using the deployment's configured recipe. Requests do not select recipes.

The response body uses the provider's OpenAI-compatible format. Reef adds the x-reef-agent-record-id response header. Its value is the receipt that a later report uses to identify this interaction. A report can contain a numeric score, textual or structured feedback, and the receipts it evaluates. This example reports both a score and a short explanation.

import os
import httpx

reef = httpx.Client(
    base_url="http://127.0.0.1:8900",
    headers={"Authorization": f"Bearer {os.environ['REEF_TOKEN']}", "x-reef-scenario": "hello-reef"},
    timeout=300,
)

# Send a provider-compatible inference request
response = reef.post(
    "/v1/chat/completions",
    json={
        "model": os.environ["MODEL_PATH"],
        "messages": [{"role": "user", "content": "Return exactly: reef is ready"}],
    },
)

response.raise_for_status()
receipt = response.headers["x-reef-agent-record-id"]
answer = response.json()["choices"][0]["message"]["content"]

# Sending report about the inference
matched = answer.strip() == "reef is ready"

reef.post(
    "/reef/report",
    json={"score": float(matched), "feedback": "matched" if matched else "wrong answer", "references": [receipt]},
).raise_for_status()

feedback carries the richer signal, plain text or a structured object, for recipes that read more than a scalar. The endpoint will validate the report schema (reef/core/reports/).

Watch it learn and grow

Once the recipe has enough feedback, it runs a training step and synchronizes the updated weights to the serving runtime. Later inference requests use the current version without restarting Reef.

Harness-evolving deployment

Improve harness skills using a model API instead of GPUs.

Pass the model at startup using REEF_UPSTREAM_MODEL; no YAML edit is needed. deployment.yaml uses this variable for both serving and evaluation. From your Reef checkout and activated Python environment, replace the model ID and API key below with your provider's values:

export REEF_UPSTREAM_URL="https://api.openai.com"  # No /v1 suffix
export REEF_UPSTREAM_MODEL="REPLACE_WITH_YOUR_PROVIDER_MODEL_ID"
export REEF_UPSTREAM_API_KEY="your-openai-api-key"
reef serve -c tutorials/evolve-your-harness/configs/deployment.yaml

Use the exact model ID accepted by your provider, not the placeholder above. An unset or empty REEF_UPSTREAM_MODEL is reported at startup.

For another provider, use its base URL, model name, and API key. This config deploys Reef on 8901 with reef-local as its access token.

In another terminal, install the harness and run a task:

export REEF_TOKEN="reef-local"   # the script writes it into the installed harness's
                                 # model binding; keep it exported for `report`
curl -fsS -H "Authorization: Bearer $REEF_TOKEN" \
  'http://localhost:8901/reef/harness/install?adapter=pi' | bash
reef-pi -p "fix the failing test in auth.py"

# After running your tests, report the actual result:
reef-pi report --score 0 --feedback "missed the empty-token case"

If you already installed the harness with your-model-name, set REEF_UPSTREAM_MODEL, stop and restart reef serve with the command above, then rerun the harness install command before retrying reef-pi. Installation writes the model ID into the local harness configuration.

Failed reports trigger a candidate skill update. Reef evaluates it against the current harness on the tutorial's three coding tasks and publishes it only if it wins. See the tutorial to customize the tasks and evaluation.

To ask for a harness change in plain words and see the whole path from the ask to the install, run the harness requests tutorial.

Recipes and examples

Choose a recipe based on your workload's feedback and the artifact you want to update. The implementations live in this repository's recipes/ cookbook, are selected by dotted class reference, and do not ship in the Reef wheel.

Workload Recipe guide Updated artifact Examples and results
A stream of tasks scored by tests or a verifier SAO Model weights Example · Results
Agent traffic with useful next-state signals and no explicit reports OpenClaw-RL Model weights Example
Repeated, scored attempts at one problem TTT-Discover Model weights Example · Results
Parallel coding agents on one task, attempts scored by a grader (CORAL) CORAL TTT Model weights Example
Scored code search with a trainable guidance model and a frozen executor Guidance-TTT / TTTD Guidance-model weights Example · Results
Agent feedback used to evolve its skill pool SkillClaw Harness skills; no training GPUs Example
Scores and transcripts used to improve prompts and instructions GEPA Harness; fixed model weights Example and results

For a small walkthrough of feedback, candidate edits, and publication, start with the coding harness tutorial. Each result page documents its task, evaluation setup, measurements, and limitations.

Architecture

Reef architecture: harness requests flow through a scenario to inference. Receipt-linked feedback feeds records and recipe training; artifact evaluation selects updates for versioned publication. Rejected candidates leave the current release serving.

Learn more

The documentation is organized in the following order:

Community & Contributing

Working on continual self-improving agent?

If Reef looks useful to you, please give it a ⭐ — it helps the community to discover and contribute to the project.

The Team

Reef brings together people exploring how agents can learn from experience and improve over time. The people below help turn that idea into working infrastructure.

This list is non-exhaustive, with team members listed alphabetically by last name:

Wenhao Chai, Shuangrui Ding, Hao He, Haoze He, Chonghe Jiang, Nan Jiang, Xuan Jiang, Xiaochen Li, Paul Liang, Bo Liu, Boyuan Long, Qiuyang Mang, Zhenting Qi, Ao Qu, Mingruo Qu, Zhaokai Wang, Xuezhi Yan, Hanfei Yu, Haofei Yu, Simon Yu, Han Zheng, Kaichen Zhou, Zijian Zhou, Jiacheng Zhu, Dingyi Zhuang.

Star History

Reef Star History Chart

Acknowledgements

We are particularly grateful to these projects which power important parts of Reef:

reef (Human-Agent-Society/reef)

Searching Database…

7.6K ·GitHub
Reef
Что это такое?Reef — это инфраструктура с открытым исходным кодом для агентов, которые непрерывно совершенствуют себя. Она объединяет инференс агента, сбор обратной связи, обучение и доставку версий, чтобы агент продолжал улучшаться по мере того, как вы с ним работаете.
Reef — это фреймворк для обучения, слой обслуживания или и то, и другое?И то, и другое: Reef обслуживает живой трафик агентов как инференс-движок и при этом управляет обучением и доставкой версий, чего обычные движки или фреймворки RL сами по себе не делают.
Как установить Reef и запустить развёртывание?Установите его из PyPI или клонируйте репозиторий для установки из исходников (это нужно для примеров обучения), используя uv для управления пакетами, затем запустите развёртывание командой 'reef serve' с конфигурацией deployment.yaml.
Что нужно установить перед запуском Reef?Нужен системный пакет git-lfs для хранения артефактов и чекпоинтов, а также либо обучаемая модель с поддерживаемым стеком GPU (для обучения весов), либо эндпоинт API модели и ключ (для эволюции обвязки агента).
Нужны ли мне GPU, чтобы пользоваться Reef?Только для пути обучения весов, который должен работать в среде, отвечающей требованиям к GPU для обучения в Slime/SGLang. Развёртывания с эволюцией обвязки агента, наоборот, нужны только API модели и никаких локальных GPU для обучения.
Я сам запускаю Reef или это облачный сервис?Reef разворачивается у вас самостоятельно: вы запускаете 'reef serve' на своей инфраструктуре, и она открывает эндпоинты, совместимые с OpenAI и Anthropic ('/v1/chat/completions' и '/v1/messages'), для трафика ваших агентов.
Какие рецепты идут в комплекте с Reef?В репозитории есть каталог recipes/ с готовыми рецептами, включая SAO, OpenClaw-RL, TTT-Discover, CORAL TTT, Guidance-TTT/TTTD, SkillClaw и GEPA; они охватывают и обучение весов модели, и эволюцию навыков обвязки агента.
Чем это отличается от простого запуска vLLM или SGLang?Движки инференса вроде vLLM или SGLang только обслуживают трафик. Reef добавляет к этому сопоставление обратной связи, триггеры обучения, управление версиями и непрерывную работу во время обновлений.
Нужно ли перезапускать Reef после того, как он чему-то научится?Нет. Как только рецепт обучает новые веса или кандидатный навык обвязки побеждает текущий в оценке, Reef публикует его, и последующие запросы автоматически используют новую версию.
Могу ли я улучшать промпты агента без обучения модели?Да. Развёртывание с эволюцией обвязки улучшает промпты, правила и навыки через эндпоинт API модели, и локальные GPU для обучения не нужны.
What are similar projects?OpenClaw-RL, verl, rrsi, ms-swift, agent-lightning, ApexAgents-SkyRL-Recipe, DeepAnalyze, NeoHorse
Which topics is it in?agent-infrastructure, self-improving-agents, llm-training, sglang, model-training, agent-harness, training, vllm, harness, agent-framework, reinforcement-learning, evaluation
Creator
Human-Agent-Society
Size
37 MB
Stars
7.6K
Updated
CommitOct 16Releasev0.1.3Oct 14
Age
1mo
Commits · 90 Days