Project_RL

Implementation of the Deep Deterministic Policy Gradient (DDPG) and Maximum A Posteriori Policy Optimization (MPO) Reinforcement Learning Algorithms for continuous control on OpenAI gym environments.

Prerequisites

To use the Algorithms will require python3 (>=3.6.5).

This should install all dependencies and the packages (cloning -> into a directory of choice):
```
git clone https://github.com/theogruner/rl_pro_telu
cd rl_pro_telu
pip install -e .
```
Or they can be installed manually

Installation gym: https://github.com/openai/gym

Installation quanser: https://git.ias.informatik.tu-darmstadt.de/quanser/clients/tree/master

Additionally PyTorch, Numpy, Tensorflow, TensorboardX are required.

Usage Examples

Note

The algorithms are intended for continuous gym environments !

DDPG

Usage with provided noises (Ornstein Uhlenbeck, Adaptive Parameter)

If you want to use DDPG with our pre-implemented noises you can simply do this via the main_ddpg.py file.
- Training on the Qube-v0 (can be replaced with any environment id e.g. Levitation-v0) environment with default hyperparameters and Ornstein-Uhlenbeck noise, saving the model as furuta_model.pt and the log in furuta_model (saves by default as ddpg_model.pt, and log in a automatic generated directory). Saving/logging could be disabled with --no-save and --no-log
```
python3 path/to/main_ddpg.py --env Qube-v0 --save_path furuta_model.pt --log_name furuta_log
```
- Loading a saved model and evaluating it in the Qube-v0 environment. Number of episodes and their length for testing can be adapted with --eval_episodes and --eval_ep_length
```
python3 path/to/main_ddpg.py --env Qube-v0 --no-train --eval --load furuta_model.pt
```
- Training and evaluating are executed sequentially, so if u want to evaluate a just trained model, this will work:
```
python3 path/to/main_ddpg.py --env Qube-v0 --train -eval --save_path furuta_model.pt --log_name furuta_log 
```
- Loading a model and setting the train flag will continue training on the loaded model
```
python3 path/to/main_ddpg.py --env Qube-v0 --train -eval --save_path furuta_model.pt --log_name furuta_log --load furuta_model.pt
```
- Adapting hyperparameters is quite intuitive:
```
python3 path/to/main_ddpg.py --env Qube-v0 --gamma 0.5 --tau 0.1 --batch_size 1024
```
- For more information, e.g. about adaptable hyperparameters, use:
```
python3 path/to/main_ddpg.py --help
```

Usage with self defined noise

To use a self defined noise you will need to write a script by yourself.

Make sure the script is the same directory as the ddpg package (only if you didn't install them with PyPI).
The noise should extend the Noise class in noise.py (contain a reset and iteration function)

Following examples cover previous examples functionality

import gym    
import quanser_robots

from ddpg import DDPG
from ddpg import OrnsteinUhlenbeck

# create environment and noise
env = gym.make('Qube-v0')
action_shape = env.action_space.shape[0] 
noise = OrnsteinUhlenbeck(action_shape)

# setup a DDPG model w.r.t. the environment and self defined noise
model = DDPG(env, noise, save_path="furuta_model.pt", log_name="furuta_log")

# load a model
model.load_model("furuta_model.pt")
#trains the model
model.train()
#evaluates the model and returns a meaned reward over all episodes
mean_reward = model.eval(episodes=100, episode_length=500)
print(mean_reward)

# always close the environment when finished 
env.cose()

Setting Hyperparameters would look something like this:

model = DDPG(env, noise, gamma=0.5, tau=0.1, learning_rate=1e-2)

Using a model as a controller

If you want to use your a model as a simple controller just call the model passing an observation:

ctrl = DDPG(env, noise)
ctrl.load('furuta_model.pt')

while not done:
   env.render()
   act = ctrl(obs)
   obs, rwd, done, info = env.step(act)

# always close the environment when finished
env.close()

MPO

Using MPO is analogous to ddpg

Use main_mpo.py instead of main_ddpg.py
For information on the parameters you can set: python3 main_mpo.py --help

Due to no noise needed writing an own script is not necessary but for completeness a little code snippet as example:

import gym
import quanser_robots
from mpo import MPO

env = gym.make('Qube-v0')
model = MPO(env, save_path="furuta_model.pt", log_name="furuta_log")
# continues like DDPG example ...

Logging

By default logging is enabled and safes the logs in the runs/ directory. Name of the logs can be set by the log_name parameter (--log_name argument). Inspecting them works with:

tensorboard --logdir=*/PATH/TO/runs*

This starts a local server, which can be accessed in the browser. Connecting to the server should result in something like this:

Open Source Infos

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Authors

Theo Gruner - theogruner
Luca Dziarski - DariusSchneider

See also the list of contributors who participated in this project.

License

This project is licensed under the GNU GPL3 License - see the LICENSE file for details

theogruner/rl_pro_telu