/lets-do-irl

Inverse RL algorithms (APP, MaxEnt, GAIL, VAIL)

Primary LanguagePythonMIT LicenseMIT

Let's do Inverse RL

image

Introduction

This repository contains PyTorch (v0.4.1) implementations of Inverse Reinforcement Learning (IRL) algorithms.

  • Apprenticeship Learning via Inverse Reinforcement Learning [2]
  • Maximum Entropy Inverse Reinforcement Learning [4]
  • Generative Adversarial Imitation Learning [5]
  • Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow [6]

We have implemented and trained the agents with the IRL algorithms using the following environments.

For reference, reviews of below papers related to IRL (in Korean) are located in Let's do Inverse RL Guide.

Table of Contents

Mountain car

We have implemented APP, MaxEnt using Q-learning as RL step in MountainCar-v0 environment.

1. Information

2. Expert's demonstrations

Navigate to expert_demo.npy in lets-do-irl/mountaincar/app/expert_demo or lets-do-irl/mountaincar/maxent/expert_demo.

Shape of expert's demonstrations is (20, 130, 3); (number of demonstrations, length of demonstrations, states and actions of demonstrations)

If you make demonstrations, Navigate to make_expert.py in lets-do-irl/mountaincar/app/expert_demo or lets-do-irl/mountaincar/maxent/expert_demo.

3. Train & Test

APP

Navigate to lets-do-irl/mountaincar/app folder.

Train the agent wtih APP without rendering.

python train.py

If you want to test APP, Test the agent with the saved model app_q_table.npy in app/results folder.

python test.py

MaxEnt

Navigate to lets-do-irl/mountaincar/maxent folder.

Train the agent wtih MaxEnt without rendering.

python train.py

If you want to test MaxEnt, Test the agent with the saved model maxent_q_table.npy in maxent/results folder.

python test.py

4. Trained Agent

We have trained the agents with two different IRL algortihms using MountainCar-v0 environment.

Algorithms Scores / Episodes GIF
APP app
MaxEnt maxent

Mujoco Hopper

We have implemented GAIL, VAIL using PPO as RL step in Hopper-v2 environment.

1. Installation

2. Expert's demonstrations

Navigate to expert_demo.p in lets-do-irl/mujoco/gail/expert_demo or lets-do-irl/mujoco/vail/expert_demo.

Shape of expert's demonstrations is (50000, 14); (number of demonstrations, states and actions of demonstrations)

We used demonstrations that get scores between about 2200 and 2600 on average.

If you want to make demonstrations, Navigate to main.py in lets-do-irl/mojoco/ppo folder.

Also, you can see detailed implementation story (in Korean) of PPO in PG Travel implementation story.

3. Train & Test

GAIL

Navigate to lets-do-irl/mujoco/gail folder.

Train the agent wtih GAIL without rendering.

python main.py

If you want to Continue training from the saved checkpoint,

python main.py --load_model ckpt_4000_gail.pth.tar
  • Note that ckpt_4000_gail.pth.tar file should be in the mujoco/gail/save_model folder.

If you want to test GAIL, Test the agent with the saved model ckpt_4000_gail.pth.tar in the mujoco/gail/save_model folder.

python test.py --load_model ckpt_4000_gail.pth.tar
  • Note that ckpt_4000_gail.pth.tar file should be in the mujoco/gail/save_model folder.

VAIL

Navigate to lets-do-irl/mujoco/vail folder.

Train the agent wtih VAIL without rendering.

python main.py

If you want to Continue training from the saved checkpoint,

python main.py --load_model ckpt_4000_vail.pth.tar
  • Note that ckpt_4000_vail.pth.tar file should be in the mujoco/vail/save_model folder.

If you want to test VAIL, Test the agent with the saved model ckpt_4000_vail.pth.tar in the mujoco/vail/save_model folder.

python test.py --load_model ckpt_4000_vail.pth.tar
  • Note that ckpt_4000_vail.pth.tar file should be in the mujoco/vail/save_model folder.

4. Tensorboard

Note that the results of trainings are automatically saved in logs folder. TensorboardX is the Tensorboard-like visualization tool for Pytorch.

Navigate to the lets-do-irl/mujoco/gail or lets-do-irl/mujoco/vail folder.

tensorboard --logdir logs

5. Trained Agent

We have trained the agents with two different IRL algortihms using Hopper-v2 environment.

Algorithms Scores / Iterations (total sample size : 2048)
PPO (to compare) ppo
GAIL gail
VAIL vail
Total total

Reference

We referenced the codes from below repositories.

Implementation team members

Dongmin Lee (project manager) : Github, Facebook

Seungje Yoon : Github, Facebook

Seunghyun Lee : Github, Facebook

Geonhee Lee : Github, Facebook