cleanrl README

The project's README from github.com/vwxyzjn/cleanrl. As one markdown file: README.md.

CleanRL (Clean Implementation of RL Algorithms)

tests docs Code style: black Imports: isort Open In Colab

CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:

You can read more about CleanRL in our JMLR paper and documentation.

Notable CleanRL-related projects:

ℹ️ Support for Gymnasium: Farama-Foundation/Gymnasium is the next generation of openai/gym that will continue to be maintained and introduce new features. Please see their announcement for further detail. We are migrating to gymnasium and the progress can be tracked in vwxyzjn/cleanrl#277.

⚠️ NOTE: CleanRL is not a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

Get started

Prerequisites:

To run experiments locally, give the following a try:

git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

# alternatively, you could use `uv venv` and do
# `python run cleanrl/ppo.py`
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000

# open another terminal and enter `cd cleanrl/cleanrl`
tensorboard --logdir runs

To use experiment tracking with wandb, run

wandb login # only required for the first time
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --track \
    --wandb-project-name cleanrltest

If you are not using uv, you can install CleanRL with requirements.txt:

# core dependencies
pip install -r requirements/requirements.txt

# optional dependencies
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-mujoco_py.txt
pip install -r requirements/requirements-procgen.txt
pip install -r requirements/requirements-envpool.txt
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-jax.txt
pip install -r requirements/requirements-docs.txt
pip install -r requirements/requirements-cloud.txt
pip install -r requirements/requirements-memory_gym.txt

To run training scripts in other games:

uv venv

# classic control
python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1

# atari
uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install ".[envpool]"
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
uv run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3

# procgen
uv pip install ".[procgen]"
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot

# ppo + lstm
uv pip install ".[atari]"
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4

You may also use a prebuilt development environment hosted in Gitpod:

Open in Gitpod

Algorithms Implemented

Algorithm Variants Implemented
✅ Proximal Policy Optimization (PPO) ppo.py, docs
ppo_atari.py, docs
ppo_continuous_action.py, docs
ppo_atari_lstm.py, docs
ppo_atari_envpool.py, docs
ppo_atari_envpool_xla_jax.py, docs
ppo_atari_envpool_xla_jax_scan.py, docs)
ppo_procgen.py, docs
ppo_atari_multigpu.py, docs
ppo_pettingzoo_ma_atari.py, docs
ppo_continuous_action_isaacgym.py, docs
ppo_trxl.py, docs
✅ Deep Q-Learning (DQN) dqn.py, docs
dqn_atari.py, docs
dqn_jax.py, docs
dqn_atari_jax.py, docs
✅ Categorical DQN (C51) c51.py, docs
c51_atari.py, docs
c51_jax.py, docs
c51_atari_jax.py, docs
✅ Soft Actor-Critic (SAC) sac_continuous_action.py, docs
sac_atari.py, docs
✅ Deep Deterministic Policy Gradient (DDPG) ddpg_continuous_action.py, docs
ddpg_continuous_action_jax.py, docs
✅ Twin Delayed Deep Deterministic Policy Gradient (TD3) td3_continuous_action.py, docs
td3_continuous_action_jax.py, docs
✅ Phasic Policy Gradient (PPG) ppg_procgen.py, docs
✅ Random Network Distillation (RND) ppo_rnd_envpool.py, docs
✅ Qdagger qdagger_dqn_atari_impalacnn.py, docs
qdagger_dqn_atari_jax_impalacnn.py, docs

Open RL Benchmark

To make our experimental data transparent, CleanRL participates in a related project called Open RL Benchmark, which contains tracked experiments from popular DRL libraries such as ours, Stable-baselines3, openai/baselines, jaxrl, and others.

Check out https://benchmark.cleanrl.dev/ for a collection of Weights and Biases reports showcasing tracked DRL experiments. The reports are interactive, and researchers can easily query information such as GPU utilization and videos of an agent's gameplay that are normally hard to acquire in other RL benchmarks. In the future, Open RL Benchmark will likely provide an dataset API for researchers to easily access the data (see repo).

Support and get involved

We have a Discord Community for support. Feel free to ask questions. Posting in Github Issues and PRs are also welcome. Also our past video recordings are available at YouTube

Citing CleanRL

If you use CleanRL in your work, please cite our technical paper:

@article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}

Acknowledgement

CleanRL is a community-powered by project and our contributors run experiments on a variety of hardware.

cleanrl (vwxyzjn/cleanrl)

Searching Database…

10.6K ·GitHub
Cleanrl
What is this?CleanRL is a collection of high-quality single-file implementations of deep reinforcement learning algorithms (PPO, DQN, C51, SAC, DDPG, TD3, PPG and more), written to be read and tweaked rather than imported.
Is it a library I can install and import into my project?No — the authors are explicit that it is not a modular library and not meant to be imported. Each algorithm variant lives in one self-contained Python file, and duplicated code is an intentional price for readability.
How do I get an experiment running?Install Python (>=3.7.1,<3.11) and uv 0.7.9+, then run one of the training scripts, e.g. ppo.py. Without uv you can install from requirements.txt, or open the prebuilt Gitpod development environment.
What is the status with Gymnasium support?The project is migrating to Gymnasium, the maintained successor of openai/gym; progress is tracked in issue #277, so environment support is still in transition.
Who should pick CleanRL over a regular RL library?People who want to understand every implementation detail of an algorithm variant, or prototype advanced features modular DRL libraries do not support — the minimal code gives great debugging without heavy subclassing.
Which algorithms and variants are implemented?PPO (with about a dozen variants: Atari, continuous actions, LSTM, multi-GPU, Isaac Gym, Procgen and more), DQN, C51, SAC, DDPG, TD3, PPG, RND and Qdagger — roughly 30 script variants in total.
How do I track and compare my training runs?Training integrates with Weights & Biases for experiment tracking, and via the related Open RL Benchmark you can browse interactive reports comparing CleanRL with Stable-Baselines3, openai/baselines and jaxrl, down to GPU utilization and gameplay videos.
Do any algorithms come in JAX versions?Yes — the algorithms table lists JAX ports next to the PyTorch ones, e.g. dqnjax.py, c51atarijax.py, ddpgcontinuousactionjax.py and td3continuousactionjax.py, plus several JAX-based PPO and Qdagger variants.
Is there a paper to cite if I use it in research?Yes — the README asks you to cite the CleanRL technical paper when you use it in your work.
What are similar projects?tianshou, reinforcement-learning, dopamine, Gymnasium, easy-rl, train-llm-from-scratch, verl, GR00T-WholeBodyControl
Which topics is it in?gym, wandb, ppo, deep-reinforcement-learning, jax, training, reinforcement-learning, pytorch, benchmark, deep-learning, research, gpu
Creator
vwxyzjn
Size
137.9 MB
Stars
10.6K
Updated
CommitApr 20Releasev1.0.0Nov 14
Age
7y
Commits · 90 Days