Reinforcement-learning foundations, worked question-first alongside CS224R (Stanford, Finn / Hausman — both Physical Intelligence co-founders). Policy gradients and value methods are what modern imitation learning and VLAs descend from.
Status: active. All three exercise skeletons are committed, and their learner TODOs remain open.
| File | What |
|---|---|
01_value_iteration.py |
Value iteration on a gridworld — the Bellman update, exactly |
02_q_learning.py |
Tabular Q-learning — model-free control from experience |
03_reinforce.py |
REINFORCE — the policy gradient at the root of ACT / Diffusion training |
Build each from scratch, then diff against the lecture's reference. Completion of these exercises and the behavior-cloning work in Chapter 3 forms the next artifact gate.
This chapter builds on stateful control loops and observable feedback, introducing reinforcement-learning value methods and policy gradients. Its artifact gate is: Completed value-iteration, Q-learning, REINFORCE, and behavior-cloning implementations with executable checks.
Next guide milestone: M2 — Learn Robot Data and Policies with LeRobot
Previous: Chapter 1 — Transformer · Dependency map · Next: Chapter 3 — Bridge to robotics · Third-party notices