/implementation-matters

Primary LanguagePythonMIT LicenseMIT

Code for "Implementation Matters in Deep RL: A Case Study on PPO and TRPO"

This repository contains our implementation of PPO and TRPO, with manual toggles for the code-level optimizations described in our paper. We assume that the user has a machine with MuJoCo and mujoco_py properly set up and installed, i.e. you should be able to run the following command on your system without errors:

import gym
gym.make_env("Humanoid-v2")

The code itself is quite simple to use. To run the ablation case study discussed in our paper, you can run the following list of commands:

  1. cd configs/
  2. mkdir PATH_TO_OUT_DIR and change out_dir to this in the relevant config file. By default agents will be written to results/{env}_{algorithm}/agents/.
  3. python {config_name}.py
  4. cd ..
  5. Edit the NUM_THREADS variables in the run_agents.py file according to your local machine.
  6. Train the agents: python run_agents.py PATH_TO_OUT_DIR/agent_configs
  7. The outputs will be in the agents subdirectory of OUT_DIR, readable with the cox python library.

See the MuJoCo.json file for a full list of adjustable parameters.