OpenClaw-RL README

The project's README from github.com/Gen-Verse/OpenClaw-RL. As one markdown file: README.md.

OpenClaw-RL Claw-RL logo

Empowering OpenClaw with RL — Train a personalized agent simply by talking to it.

Scalable RL in real-world settings — Agentic RL for terminal, GUI, SWE, and tool-call settings.

Fully Async Zero API or Zero GPU Personalized Auto Language Feedback Hybrid RL General Agentic RL

Tech Report OpenClaw-RL Blog OpenClaw Plugin Slime Based Tinker Supported License Apache 2.0

📰 News

💡 TL;DR

OpenClaw-RL is a fully asynchronous reinforcement learning framework that turns everyday conversations into training signals for personalized AI agents, and supports training general agents with large-scale environment parallelization.

Most RL-for-LLM systems assume centralized, batch-mode training with pre-collected datasets. OpenClaw-RL takes a fundamentally different approach: it wraps your self-hosted model in OpenClaw as an OpenAI-compatible API, intercepts live multi-turn conversations, and continuously optimizes the policy in the background — all without interrupting your usage.

Overview

Highlights: Fully async 4-component loop · Self-hosted & private · Zero manual labeling · Three learning paradigms (Binary RL / OPD / Combine) · Personal + General agent support

🌈 Features

Fully Asynchronous 4-Component Architecture

OpenClaw-RL decouples agent serving, rollout collection, PRM/judge evaluation, and policy training into independent async loops. None of them block one another: the model continues serving requests while training runs in the background, and judging happens concurrently with new interactions.

Self-Hosted & Private by Design

The entire stack, including the policy model, judge/PRM, and trainer, runs on your own infrastructure. Conversation data stays within your system, and no third-party model API is required.

From Feedback to Gradient — Automatically

You do not need to manually label data. The system automatically:

  • Organizes multi-turn interactions into session-aware training trajectories
  • Classifies API messages into main-line (trainable) vs. side (non-trainable) turns
  • Uses the next user, environment, or tool feedback as a natural "next-state" signal
  • Runs PRM/judge evaluation asynchronously, with majority voting when needed for more robust scoring
  • Submits ready samples to the trainer as they become available

Three Optimization Methods in One Framework

Binary RL (GRPO): A Process Reward Model scores each turn based on next-state feedback. The scalar reward is then used with GRPO advantage estimation and a PPO-style clipped surrogate loss.

On-Policy Distillation (OPD): When the next state reveals useful hindsight, a judge model extracts a textual hint. This hint augments the original prompt to create an enhanced teacher, whose token-level log-probability gap with the student becomes a directional advantage signal richer than any scalar reward.

Hybrid Method: OpenClaw-RL further combines Binary RL and OPD in a unified training recipe, leveraging the dense scalar supervision of Binary RL together with the richer token-level directional signal from OPD. This combination achieves stronger and more robust optimization than either method alone.

From Personal Agents to Real-World Agentic RL

The same framework supports both personalized OpenClaw optimization and scalable RL for terminal, GUI, SWE, and tool-call agents in real-world settings.

🎯 Roadmap

Our long-term goal is to advance personalized, practically useful agents with reinforcement learning. The roadmap has two tracks:

Track 1 — Personal Agent Optimization (Small-Scale but Personal)

✅ Release Track 1: Fully async OpenClaw-RL framework with Binary RL + OPD
✅ Best recipe discovery via demonstration experiments
✅ Support LoRA Training
✅ Deploy training on Tinker
✅ Deploy training on Fireworks AI

Track 2 — General Agents Optimization (Scalable Infra)

✅ Release Track 2: Scalable agentic RL infra for general agents
✅ Support Qwen3.5
⬜ Support more cloud services

📝 Contents

🔧 Personal Agent Optimization Quick Start

1. Deployment Requirements

For detailed environment setup, see Slime or ./instructions/README.md.

2. Start the RL Server

cd slime
bash ../openclaw-combine/run_qwen3_4b_openclaw_topk_select.sh

This method combines binary RL and OPD to achieve the best optimization.

See ./openclaw-combine/README.md for algorithm details.

Once running, the model is served as an OpenAI-compatible API at:

http://<HOST_IP>:30000/v1

where <HOST_IP> is the IP address of the machine running the RL server (e.g. 115.190.98.251). The port 30000 is the default and can be changed via the PORT environment variable.

Take note of this endpoint — you will need it when configuring OpenClaw in the next step.

We also provide an interesting case for evaluation. A student who uses OpenClaw to do homework, does not want to be found using AI. A teacher who also uses OpenClaw to grade student's homework, wants the comments to be specific and friendly.

Evaluation Setting — Both student, TA and teacher use AI! Overview

See ./openclaw-test/README.md for setup and algorithm details. Example of evaluation results.

3. OpenClaw Setup

You can use your own openclaw, just install this extension.

Then configure OpenClaw to route requests to your RL server.

Open your openclaw.json (or the equivalent settings file) and add a provider entry under "models" → "providers":

Example of Slime-based RL server:

{
  "models": {
    "providers": {
      "qwen": {
        "baseUrl": "http://<HOST_IP>:30000/v1",
        "apiKey": "apiKey",
        "api": "openai-completions",
        "models": [
          {
            "id": "qwen3-4b",
            "name": "Qwen3 4B",
            "reasoning": true,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 32768,
            "maxTokens": 8192
          }
        ]
      }
    }
  }
}

Replace <HOST_IP> with the IP address of your RL server machine. The apiKey should match the SGLANG_API_KEY you set when starting the server.

Example of Tinker-based RL server:

{
  "models": {
    "providers": {
      "openclaw-rl": {
        "baseUrl": "http://localhost:30000/v1",
        "apiKey": "no-auth-needed",
        "api": "openai-completions",
        "models": [
          {
            "id": "qwen3-4b-lora",
            "name": "Qwen3 4B (OpenClaw-RL LoRA)",
            "reasoning": true,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 32768,
            "maxTokens": 8192
          }
        ]
      }
    }
  }
}

That's it — start chatting with your OpenClaw agent. The RL server will automatically collect conversation trajectories, compute rewards, and train the model. Your agent gets better the more you use it.

🔧 Agentic RL in Real-world Settings

The same asynchronous RL backbone that powers our personal-agent setting can also support large-scale optimization for these broader real-world environments.

Setting Environment Next-state signal Horizon
Terminal Shell execution sandbox stdout/stderr, exit code Long
GUI Screen state + accessibility tree Visual state diff, task progress Long
SWE Code repository + test suite Test verdicts, diff, lint output Long
Tool-call API/function execution Return values, error traces Medium

🖥️ Terminal Agent — the most widely used computer-use agent

cd slime
bash ../terminal-rl/terminal_qwen3_8b_rl.sh

See ./terminal-rl/README.md for setup details.

📟 GUI Agent — the most general computer-use agent

cd slime
bash ../gui-rl/gui_qwen3vl_8b_rl.sh

See ./gui-rl/README.md for setup details.

👨‍💻 SWE Agent — software engineering agent

cd slime
bash ../swe-rl/run_swe_rl_32b_remote_8nodes.sh

See ./swe-rl/README.md for setup details.

🛠️ Tool-call Agent — the most practical agent

cd slime
bash ../toolcall-rl/retool_qwen3_4b_rl.sh

See ./toolcall-rl/README.md for setup details.

📖 Citation

@article{wang2026openclawrl,
  title={OpenClaw-RL: Train Any Agent Simply by Talking},
  author={Wang, Yinjie and Chen, Xuyang and Jin, Xiaolong and Wang, Mengdi and Yang, Ling},
  journal={arXiv preprint arXiv:2603.10165},
  year={2026}
}

@article{wang2026rlanything,
  title={RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System},
  author={Wang, Yinjie and Xie

OpenClaw-RL (Gen-Verse/OpenClaw-RL)

Searching Database…

5.7K ·GitHub
OpenClaw-RL
Что это такое?OpenClaw-RL — полностью асинхронный фреймворк обучения с подкреплением, который превращает повседневные разговоры с агентом в сигналы для обучения. Так можно обучить персонализированного агента OpenClaw, просто общаясь с ним, или обучить универсальных агентов для задач в терминале, GUI, SWE и вызовов инструментов.
Это готовый агент или фреймворк, к которому я подключаю свой агент?Это фреймворк для обучения, а не самостоятельный агент: он оборачивает вашу собственную модель в API, совместимый с OpenAI, и запускает четыре независимых асинхронных цикла (раздача запросов, сбор траекторий, оценка, обучение политики) поверх базового RL-фреймворка Slime.
Как заставить это обучать моего агента?Настройте окружение по инструкции ./instructions/README.md (CUDA 12.9, Python 3.12, Slime), запустите RL-сервер, чтобы ваша модель была доступна как API, совместимый с OpenAI, затем добавьте этот эндпоинт как провайдера в openclaw.json вашего OpenClaw. Сервер собирает ваши разговоры и обучается в фоне, пока вы общаетесь с агентом.
Нужен ли мне полноценный кластер GPU, чтобы это запустить?По умолчанию да: 8 GPU (настраивается через NUM_GPUS, ACTOR_GPUS, ROLLOUT_GPUS, PRM_GPUS), а также CUDA 12.9, Python 3.12 и фреймворк Slime RL. Либо можно обойтись без локальных GPU и запускать через облачные бэкенды, например Tinker или Fireworks AI.
В чём подвох у настройки в одну строку?Базовая настройка личного агента требует серьёзного железа (8 GPU) или платного облачного бэкенда. Кроме того, это проект на стадии исследований (v1 вышла в феврале 2026), построенный на конкретной базе, Slime, поэтому для обучения вы привязаны к этому стеку.
Покидают ли данные моих разговоров мой компьютер во время обучения?Нет — модель политики, судья/PRM и обучающий модуль работают на вашей собственной инфраструктуре, и API сторонней модели не требуется.
Чем отличаются Binary RL, OPD и гибридный метод?Binary RL (GRPO) оценивает каждый ход с помощью модели вознаграждения и применяет обновления GRPO в духе PPO. OPD — модель-судья формирует подсказку задним числом, которая порождает улучшенный ответ учителя, и разница на уровне токенов между учителем и учеником становится сигналом для обучения. Гибридный метод объединяет оба сигнала и даёт более сильную оптимизацию, чем каждый из них по отдельности.
Что такое OpenClaw в этом контексте?OpenClaw — это самостоятельно развёрнутый ИИ-агент, которого обучают. Вы устанавливаете расширение, чтобы ваш OpenClaw направлял свой трафик через RL-сервер, и тот учится на каждом разговоре.
Какие типы агентов, кроме личного чата, поддерживаются?Track 2 расширяет ту же асинхронную RL-основу на терминальных агентов (песочницы оболочки), GUI-агентов (состояние экрана и дерево доступности), SWE-агентов (репозитории кода с наборами тестов) и агентов вызова инструментов (выполнение API и функций).
Проверено ли это за пределами самого репозитория?Да: команда опубликовала технический отчёт, который занял первое место в Daily Papers на Hugging Face. Кроме того, фреймворк теперь поддерживает модели Qwen3.5 (4B/9B/27B, как текстовые, так и мультимодальные).
What are similar projects?MetaClaw, halo, reef, ms-swift, agent-lightning, verl, ART, minimind
Which topics is it in?openclaw-skills, grpo, llm-training, rlhf, sglang, coding, lora, async, openclaw, agent-framework, reinforcement-learning, agent-memory
Creator
Gen-Verse
Size
50.8 MB
Stars
5.7K
Updated
CommitMay 23
Age
7mo
Commits · 90 Days