AlphaZero Practical Example Part 2: Building a Training Pipeline

Live recording of online meeting reviewing Reinforcement Learning Concepts. Since AlphaZero's release, the authors have made several improvements with better theoretical guarantees. In this meeting we continue the discussion of the version of neural network enhanced Monte Carlo Tree Search described in this paper: Danihelka, I., Guez, A., Schrittwieser, J., & Silver, D. (2022). Policy Improvement by Planning with Gumbel (ICLR 2022). https://openreview.net/forum?id=bERaN... Last meeting (see    • AlphaZero Practical Example Part 1: Realti...  ) we showed how to implement the search function guided by an existing policy and value function. The result should be an improved policy whose strength improves with the number of simulations. This meeting we show how search can be used as a data collection method for supervised learning on a replay buffer. The trained policy will mimic the behavior of the enhanced search policy. From this premise, we build a training pipeline that simultaneously collects data while pulling minibatches from the buffer and updates training parameters. We discuss the key hyperparameters of this algorithm including the ratio of data collection rate to training rate. Using the same tic-tac-toe example, we demonstrate mastering this environment against a fixed opponent starting from a randomly initialized policy. You can find the notebook used to demonstrate MCTS here: https://github.com/jekyllstein/Reinfo... I'm using the following repository to store notes and interactive tools for multi-agent reinforcement learning: https://github.com/jekyllstein/MARL_c... My previous material on reinforcement learning contains complete notes on the Sutton and Barto RL book: https://jekyllstein.github.io/Reinfor... The textbook website contains materials provided by the authors including a pdf of the text, slides, and a github repository with code. MARL textbook website: https://www.marl-book.com/ MARL kickoff slides: https://docs.google.com/presentation/... This online meeting is hosted through https://www.meetup.com/boulderdatasci... and https://www.meetup.com/silicon-valley... For background material covering traditional reinforcement learning see the following playlist:    • Reinforcement Learning Tutorial Meetings   Previous meetings have covered the textbook "Reinforcement Learning: An Introduction" by Richard S. Sutton and Andrew G. Barto and the following links relate to that material and my notes/code based on it. Sutton and Barto Textbook: http://incompleteideas.net/book/the-b... HTML Notes: https://jekyllstein.github.io/Reinfor... GitHub Repository: https://github.com/jekyllstein/Reinfo... Notes and interactive tools seen in those video use the Julia Language (https://julialang.org/) and the package Pluto.jl (https://plutojl.org/). #reinforcementlearning #education #multiplayergames