PrefNet: Inverse Reinforcement Learning Approach for Robot Locomotion under Suboptimal Learning Conditions
Bachelor's Thesis in Robotics
Installation
We recommend to install using Anaconda. Afterwards, you need to perform to steps to use this repository.
- Create conda environment
conda env create -f environment.yml # environment.lock.yml for exact reproduction
- Activate environment
conda activate snake_env
Furthermore, we use the Mujoco physics engine for the simulation. You will need to create a folder called .mujoco
in your home directory and put your license key there. Also download mujoco200 or mjpro150 here and unzip it inside .mujoco
, both versions should work.
If you encounter any problem, file an issue or have a look at https://github.com/openai/mujoco-py.
You might also need to set some system variables in .bashrc/.zshrc:
# example for mjpro150, mujoco 200 analogously
export MUJOCO_PY_MJPRO_PATH=$HOME/.mujoco/mjpro150
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:~/.mujoco/mjpro150/bin
Content
This repository contains the source code to reproduce the results for my thesis. All hyperparameters are set in the configs file. Make sure that you reference the configs file when executing a python script.
Additionally, my thesis and my final presentation are stored here.
Usage
Download the data here. Overview of the folder:
├── agent <- contains the index and data files of the policy that were used to create the training data.
│ ├── 000005000.data-00000-of-00001 The policy was checkpointed every figth episode, which equals 5000 timesteps.
│ ├── 000005000.index The policy was trained for 1.5 Mio timesteps, thus this directory contains 300 policies of varying quality.
│ ├── ... If you run create dataset, it will create one trajectory of length 1000 for every saved policy.
│ ├── 001500000.data-00000-of-00001 The standard configuration will save the first 200 of those into train.npy and the other 100 into extrapolate.npy
│ └── 001500000.index
├── train.npy <- contains policy rollouts of length 1000 for the first 200 policies
├── extrapolate.npy <- contains policy rollouts of length 1000 for the last 100 policies, used for testing
└── subtrajectories.npy <- the actual training data, created from train.npy
Data Creation
subtrajectories.npy
is the actual training data. You can use it directly and skip the rest of this section.
If you want to create the data yourself, you can run create_dataset.py
. Then, you need to adjust two parameters in the create_dataset
stage in the configs.yml
file.
The model_dir
parameter is the path to the agent files you downloaded. The data_dir
parameter is the location the generated subtrajectories.npy
gets saved.
python pref_net/src/create_dataset.py --path_to_configs PATH/TO/CONFIGS
Preference Learning
In the train
stage in the configs.yml
file, you can set the parameters for the reward learning and the policies you could train on top of it.
First, define the data_dir
parameter which points to the subtrajectories.npy
file. save_dir
represents the directory the reward model and policies trained with it are stored. Concerning monitoring, tensorboard_dir
is the directory for the tensorboard logs. Lastly, ranking_method
is the ranking method the network is trained with. You can choose between Pair, Triplet and NaiveTriplet.
python pref_net/src/train.py --path_to_configs PATH/TO/CONFIGS
Author
Andreas Binder, Information Systems (B.Sc.), Technical University Munich