czp16/cde-offline-rl

Learning from Sparse Offline Datasets via Conservative Density Estimation (ICLR 2024)

Learning from Sparse Offline Datasets via Conservative Density Estimation

This project provides the open source implementation of the CDE in the paper: "Learning from Sparse Offline Datasets via Conservative Density Estimation"

Installation

1. We recommend to use Anaconda or Miniconda to manage python environment.
2. Install mujoco and mujoco-py, your can either refer to https://github.com/openai/mujoco-py or run

   wget https://mujoco.org/download/mujoco210-linux-x86_64.tar.gz
   tar -xvf mujoco210-linux-x86_64.tar.gz
   mkdir .mujoco
   mv mujoco210 ~/.mujoco/mujoco210

   It is also necessary to add below to ~/.bashrc:

   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.mujoco/mujoco210/bin
   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia

   We have included mujoco-py in requirements.txt but you may need to install libglew-dev, patchelf when compiling the mujoco-py after the installation:

   sudo apt-get install libglew-dev
   sudo apt-get install patchelf

3. Create conda env:

   cd cde-offline-rl
   conda env create -f environment.yaml
   conda activate cde

4. Install PyTorch according to your platform and cuda version.
5. Install D4rl from https://github.com/Farama-Foundation/D4RL.

Training

To run a single experiment, take maze2d-medium-v1 for example, run

python run_cde.py --env_name "maze2d-medium-v1" --hyperparams 'hyper_params/cde/maze2d.yaml' --cudaid 0 --seed 100

where --hyperparams specifies the hyperparameter files in directory ./hyper_params/, --cudaid specifies which gpu will be used for training (the defaulted -1 means using cpu).

If you want to run multiple experiments, we have also included some other training commands in bash file run_exp.sh. You can consider using & to run the commands in parallel.