jdubkim/explainable-hierarchical-planning

Final Year Project - Explainable model-based hierarchical reinforcement learning

Explainable Reinforcement Learning: A Hierarchical Planning Approach

This is based on the official implementation of the Director algorithm in TensorFlow 2.

Running the Agent

Follow the instructions below.

Install dependencies:

pip install -r requirements.txt

Train agent:

python embodied/agents/director/train.py \
  --logdir ~/logdir/$(date +%Y%m%d-%H%M%S) \
  --configs dmc_vision \
  --task dmc_swingup

See agents/director/configs.yaml for available flags and embodied/envs/__init__.py for available envs.

Some of the Results of Our Work

Our changes enable Director to perform well on DM Control Suite - SwingUp task.

Left: Director, Right: Our agent

The below is a visualisation of internal goals.

How does Director work?

Director is a practical and robust algorithm for hierarchical reinforcement learning. To solve long horizon tasks end-to-end from sparse rewards, Director learns to break down tasks into internal subgoals. Its manager policy selects subgoals that trade off exploratory and extrinsic value, its worker policy learns to achieve the goals through low-level actions. Both policies are trained from imagined trajectories predicted by a learned world model. To support the manager in choosing realistic goals, a goal autoencoder compresses and quantizes previously encountered representations. The manager chooses its goals in this compact space. All components are trained concurrently.

For more information:

• Google AI Blog
• Project website
• Research paper