📣 Announcement
We sincerely thank everyone for your support of this tutorial! A new version is coming soon. Many sections are still being organized and refined, so we appreciate your patience. Suggestions and feedback are always welcome!
News
Note: This course was created with AI assistance and has not yet been fully reviewed. It may contain factual mistakes or code that does not run as expected. Issues and pull requests are very welcome.
- [2026-08-19] 🎮 Online Classic RL Environments and Scripts: Over the past two weeks, we have added and refined a collection of online reinforcement learning environments, training scripts, and companion notebooks. Learners can now run classic reinforcement learning experiments online, inspect training logs and evaluation results, and study the algorithms more conveniently. We also fixed many previously reported bugs in the course content, links, and experiment code.
- [2026-05-15] 📖 Full English Translation & PDF Release: Complete English translation of all chapters is now available. PDF builds for both Chinese and English editions are released automatically via CI.
- [2026-05-13] 🚀 Major Upgrade: LLM and Traditional RL Hands-on Labs: Added reproducible training examples for Agentic RL (Deep Research / rLLM) and Traditional RL (Actor-Critic continuous control). Includes complete code and fine-tuning analysis for building an Agentic training system from scratch, along with new VLM RL (GeoQA geometry reasoning) hands-on experiments!
- [2026-05-02] Initial browsable open-source release for testing and feedback.
Online Training Notebooks
WalkingLab is collaborating with ModelScope to provide online training environments for classic reinforcement learning experiments. A ModelScope Studio brings the experiment interface, runtime, and training entry point together on one page, so learners can start training in a browser and observe the agent without first configuring a local environment.
Every Studio has a companion notebook under code/online-experiments. The notebook imports the same training runtime as the Studio, exposes the experiment parameters, prints the full training log, plots checkpoint evaluations, and displays the learned-policy replay or result artifact.
CPU entries run on an ordinary notebook instance. Experiments 03, 08, 10, 11, and 12 require a scheduled ModelScope xGPU Notebook and check CUDA before training.
Book Features
One continuous learning path
Begin with a CartPole trial and progress through value learning, policy optimization, and modern agents.
|
Equations meet code
Key PPO, DPO, and GRPO derivations sit beside their implementations, with every tensor accounted for.
|
Claims tested by experiments
Real training curves, ablations, and failure signals show when an algorithm works and when it does not.
|
Classic RL to language models
Policy gradients and PPO lead naturally into RLHF, DPO, GRPO, and RLVR.
|
Agents as sequential decisions
Tool use, browser interaction, and code repair become problems of states, actions, trajectories, and credit assignment.
|
Phenomena before abstractions
CartPole, LunarLander, Atari, and LLM experiments pose the problem before introducing the mathematics.
|
[!NOTE]
We hope this open course gives more learners the courage to climb toward the frontier of intelligence and solve more of the hard problems on the path to AGI.
The course is evolving quickly. We recommend focusing on chapters that are not marked as under construction; chapters still in progress may contain mistakes, and corrections or suggestions are welcome.
Help Wanted
Because compute resources are limited, we are seeking GPU support. If you can help with GPU access, please contact [email protected].
Contents
About This Book
Reinforcement learning studies a simple but difficult problem: a system acts, observes the consequences, and uses them to improve its next action. When rewards arrive late, observations are incomplete, and each update changes the distribution of future experience, the familiar supervised-learning model of inputs and labels is no longer enough. We need a language for interaction, a way to estimate long-term return, and methods that improve a policy stably from limited data.
Hands-On Modern RL follows that problem from beginning to end. CartPole and multi-armed bandits first make states, actions, rewards, and policies observable. Markov decision processes, value functions, and Bellman equations then provide a common language. From there, the book develops DQN, policy gradients, actor-critic methods, PPO, continuous control, and offline reinforcement learning. With those foundations in place, RLHF, DPO, GRPO, and RLVR become extensions of the same sequential-decision framework rather than an isolated collection of acronyms.
The second half expands the environment to tools, browsers, code repositories, vision, and audio. An action may be a passage of text, a function call, or a sequence of interface operations. A reward may come from human preference, a rule-based verifier, or a process reward model. The setting changes, but three questions run through the entire book: How should the decision process be represented? How should an outcome be credited to earlier actions? How can we tell whether a policy has actually improved?
How the Material Is Taught
Each chapter follows a problem–method–experiment–reflection rhythm. A concrete task first exposes the difficulty. The concepts and equations needed to solve it come next. Runnable code, training curves, and evaluation metrics then test the argument. The chapter closes by examining assumptions, failure modes, and the limits of the method. Mathematics explains observed behavior; experiments check the mathematics.
Implementations retain the visible skeleton of each algorithm. Readers can trace trajectory collection, advantage estimation, loss construction, and metric changes—and see how reward hacking, KL drift, entropy collapse, distribution shift, or evaluation leakage can invalidate an apparently successful run.
Who Should Read It
This book is for students, researchers, and engineers with basic machine-learning experience who want a systematic understanding of modern reinforcement learning. Readers should be comfortable with Python and basic PyTorch, and should know introductory linear algebra, probability, and calculus. The mathematical appendices rebuild these tools to the depth required by the chapters, so a separate advanced-mathematics sequence is not a prerequisite.
After completing the core chapters and labs, you should be able to:
- formulate a new decision problem using MDPs, value functions, Bellman equations, and credit assignment;
- implement, read, and diagnose DQN, REINFORCE, actor-critic methods, PPO, DPO, and GRPO;
- explain how SFT, reward modeling, preference optimization, RLHF, and RLVR fit together in LLM post-training;
- design trajectories, rewards, training loops, and evaluation protocols for tool-use, code, and multimodal agents;
- identify failure modes behind training curves and test proposed improvements with controlled experiments.
Current Status
This repository is an active courseware project. Content is being expanded chapter by chapter, with emphasis on correctness, runnable examples, and a stable learning path.
Issues and pull requests are welcome for typo fixes, conceptual corrections, reproducibility improvements, references, and focused course extensions.
Roadmap
The course is under active development. Planned milestones:
- 2026-05-02: Initial open-source browsable release for community testing and feedback.
- 2026-05-10: Publish a first stable minor version, fix early typos, and stabilize Part 1 and Part 2 content and code.
- Late May 2026: Improve reproducible LLM RL experiments and add a full RLVR hands-on module with evaluation.
- Early June 2026: Deliver Agentic RL projects step by step, from single-tool use to complex Deep Research trajectory synthesis.
- Late June 2026: Add Unity-based embodied RL environments and trainable project examples.
- July 2026 and later: Expand multimodal frontier content with full VLM RL or Diffusion RL hands-on cases.
Structure of the Book
The book contains seven parts and twenty-six chapters. Parts I–III establish the common language and algorithmic foundations of reinforcement learning. Part IV brings those tools into LLM post-training. Parts V and VI study what changes when the action space expands to tools and multimodal environments. Part VII asks how to detect failures, build trustworthy evaluations, and move the research frontier forward. The appendices provide implementation, mathematics, and engineering references.
Prologue: From Trial and Error to Modern Agents
Part I: The Language of Sequential Decisions
The book first makes an agent's failures and improvements observable, then develops the mathematical objects needed to describe long-term decisions.
| Ch. |
Topic |
Through line |
| 1 |
Starting with CartPole |
Use states, actions, rewards, policies, and training curves to see a complete RL loop. |
| 2 |
RL Problems and Definitions |
Move from exploration and exploitation to MDPs, returns, trajectories, and partial observability. |
| 3 |
Value Functions and Bellman Equations |
Express how a present action changes the future through state values, action values, and recursion. |
| 4 |
Classical RL Methods |
Compare dynamic programming, Monte Carlo, and tempor |
Hands-On Modern RL 是一门开源课程兼图书，带领学习者从基础的强化学习概念出发，经过大语言模型对齐（RLHF、DPO、GRPO、RLVR），一直学到高级智能体系统，理论配有可运行的代码。