Examples of published reinforcement learning algorithms in recent literature implemented in TensorFlow. Most of my research is in the continuous domain, and I haven't spent much time testing these in discrete domains such as Atari etc.
BipedalWalker-v2 solved using DPPO with a LSTM layer. CarRacing-v0 solved using PPO with a joined actor-critic network
Thanks to DeepMind and OpenAI for making their research openly available. Big thanks also to the TensorFlow community.
- GAE was used in all algorithms except for DPPG
- Where possible, I've added an LSTM layer to the policy and value functions. This sometimes achieved higher scores in some environments, but can have stability issues
- In some environments, having a joint network for the actor & critic performs better (i.e. where CNNs are used).
These scripts are suffixed, e.g.
ppo_joined.py
All the Python scripts are written as standalone scripts (but share some common functions in utils.py).
Just run them directly in your IDE. Or in a terminal using the -m flag:
rl-examples$ python3 -m ppo.ppo_joined
The models and TensorBoard summaries are saved in the same directory as the script. DPPO has a helper script to set off the worker threads:
rl-examples$ sh dppo/start_dppo.sh
- Python 3.6+
- OpenAI Gym 0.10.3+
- TensorFlow 1.11
- Numpy 1.13+
DPPO was tested on a 16 core machine using CPU only, so the helper script will need to be updated for your particular setup. For my setup, there was usually no speed advantage training BipedalWalker on the CPU vs GPU (GTX 1080), but CarRacing did get a performance boost due to the usage of CNN layers
- Work needed to find the correct parameters for PPO in discrete action spaces for Atari
- The LSTM batching in A3C is incorrect. Need to fix this (see
ppo_lstm.pyfor the correct implementation) - Distributed Proximal Policy Optimisation with the LSTM (
dppo_lstm.py) is sometimes a bit unstable, but does work at low learning rates