ZorroStardust/lets-do-irl

Inverse RL algorithms (APP, MaxEnt, GAIL, VAIL)

Let's do Inverse RL

Introduction

This repository contains PyTorch (v0.4.1) implementations of Inverse Reinforcement Learning (IRL) algorithms.

• Apprenticeship Learning via Inverse Reinforcement Learning [2]
• Maximum Entropy Inverse Reinforcement Learning [4]
• Generative Adversarial Imitation Learning [5]
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow [6]

We have implemented and trained the agents with the IRL algorithms using the following environments.

• OpenAI GYM Mountain car
• Mujoco Hopper

For reference, reviews of below papers related to IRL (in Korean) are located in Let's do Inverse RL Guide.

• [1] AY. Ng, et al., "Algorithms for Inverse Reinforcement Learning", ICML 2000.

• [2] P. Abbeel, et al., "Apprenticeship Learning via Inverse Reinforcement Learning", ICML 2004.

• [3] ND. Ratliff, et al., "Maximum Margin Planning", ICML 2006.

• [4] BD. Ziebart, et al., "Maximum Entropy Inverse Reinforcement Learning", AAAI 2008.

• [5] J. Ho, et al., "Generative Adversarial Imitation Learning", NIPS 2016.

• [6] XB. Peng, et al., "Variational Discriminator Bottleneck. Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow", ICLR 2019.

Table of Contents

• Let's do Inverse RL
  • Introduction
  • Table of Contents
  • Mountain car
    • 1. Information
    • 2. Expert's demonstrations
    • 3. Train & Test
      • APP
      • MaxEnt
    • 4. Trained Agent
  • Mujoco Hopper
    • 1. Installation
    • 2. Expert's demonstrations
    • 3. Train & Test
      • GAIL
      • VAIL
    • 4. Tensorboard
    • 5. Trained Agent
  • Reference
  • Implementation team members

Mountain car

We have implemented APP, MaxEnt using Q-learning as RL step in MountainCar-v0 environment.

1. Information

• Mountain car Wiki

2. Expert's demonstrations

Navigate to expert_demo.npy in lets-do-irl/mountaincar/app/expert_demo or lets-do-irl/mountaincar/maxent/expert_demo.

Shape of expert's demonstrations is (20, 130, 3); (number of demonstrations, length of demonstrations, states and actions of demonstrations)

If you make demonstrations, Navigate to make_expert.py in lets-do-irl/mountaincar/app/expert_demo or lets-do-irl/mountaincar/maxent/expert_demo.

3. Train & Test

APP

Navigate to lets-do-irl/mountaincar/app folder.

Train the agent wtih APP without rendering.

python train.py

If you want to test APP, Test the agent with the saved model app_q_table.npy in app/results folder.

python test.py

MaxEnt

Navigate to lets-do-irl/mountaincar/maxent folder.

Train the agent wtih MaxEnt without rendering.

python train.py

If you want to test MaxEnt, Test the agent with the saved model maxent_q_table.npy in maxent/results folder.

python test.py

4. Trained Agent

We have trained the agents with two different IRL algortihms using MountainCar-v0 environment.

Algorithms	Scores / Episodes	GIF
APP
MaxEnt

Mujoco Hopper

We have implemented GAIL, VAIL using PPO as RL step in Hopper-v2 environment.

1. Installation

• Mac OS (in Korean)
• Ubuntu

2. Expert's demonstrations

Navigate to expert_demo.p in lets-do-irl/mujoco/gail/expert_demo or lets-do-irl/mujoco/vail/expert_demo.

Shape of expert's demonstrations is (50000, 14); (number of demonstrations, states and actions of demonstrations)

We used demonstrations that get scores between about 2200 and 2600 on average.

If you want to make demonstrations, Navigate to main.py in lets-do-irl/mojoco/ppo folder.

Also, you can see detailed implementation story (in Korean) of PPO in PG Travel implementation story.

3. Train & Test

GAIL

Navigate to lets-do-irl/mujoco/gail folder.

Train the agent wtih GAIL without rendering.

python main.py

If you want to Continue training from the saved checkpoint,

python main.py --load_model ckpt_4000_gail.pth.tar

• Note that ckpt_4000_gail.pth.tar file should be in the mujoco/gail/save_model folder.

If you want to test GAIL, Test the agent with the saved model ckpt_4000_gail.pth.tar in the mujoco/gail/save_model folder.

python test.py --load_model ckpt_4000_gail.pth.tar

• Note that ckpt_4000_gail.pth.tar file should be in the mujoco/gail/save_model folder.

VAIL

Navigate to lets-do-irl/mujoco/vail folder.

Train the agent wtih VAIL without rendering.

python main.py

If you want to Continue training from the saved checkpoint,

python main.py --load_model ckpt_4000_vail.pth.tar

• Note that ckpt_4000_vail.pth.tar file should be in the mujoco/vail/save_model folder.

If you want to test VAIL, Test the agent with the saved model ckpt_4000_vail.pth.tar in the mujoco/vail/save_model folder.

python test.py --load_model ckpt_4000_vail.pth.tar

• Note that ckpt_4000_vail.pth.tar file should be in the mujoco/vail/save_model folder.

4. Tensorboard

Note that the results of trainings are automatically saved in logs folder. TensorboardX is the Tensorboard-like visualization tool for Pytorch.

Navigate to the lets-do-irl/mujoco/gail or lets-do-irl/mujoco/vail folder.

tensorboard --logdir logs

5. Trained Agent

We have trained the agents with two different IRL algortihms using Hopper-v2 environment.

Algorithms	Scores / Iterations (total sample size : 2048)
PPO (to compare)
GAIL
VAIL
Total

Reference

We referenced the codes from below repositories.

• Implementation of APP
• Implementation of MaxEnt
• Pytorch implementation for Policy Gradient algorithms (REINFORCE, NPG, TRPO, PPO)
• Pytorch implementation of GAIL

Implementation team members

Dongmin Lee (project manager) : Github, Facebook

Seungje Yoon : Github, Facebook

Seunghyun Lee : Github, Facebook

Geonhee Lee : Github, Facebook