simmodel

Simulation environment for Reinforcement Learning experiments in Search & Pursuit-Evasion on Graphs. Thesis work `Predicting plausible escape routes using reinforcement learning and graph representations' for UvA/AI MSc program (Thesis Proposal, Final Thesis)

Installation

Clone this repo

git clone git@github.com:rvdweerd/simmodel.git
cd simmodel

Create conda environment from yml file (check/adjust cudatoolkit version)
```
conda env create -f environment.yml
conda activate rl
```

Use pip to install `Pytorch Geometric (check/adjust cudatoolkit version)

pip install torch-scatter -f https://data.pyg.org/whl/torch-1.11.0+cu102.html
pip install torch-sparse -f https://data.pyg.org/whl/torch-1.11.0+cu102.html
pip install torch-geometric

Install stable-baselines3 contrib

cd ..
git clone https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/
cd stable-baselines3-contrib
pip install  -e .
cd ../simmodel

Training and testing models

Replicate the LSTM experiment with the PPO-GNN-LSTM model (section 'Effect of LSTM positioning', Appendix E2)
```
python Phase3_lstm-gnn-ppo_simp.py --train_on MemTask-U1 --train True --demoruns True --obs_mask freq --obs_rate 0.2 --lstm_type EMB
```
Before training, user interaction (demoruns) with the training graphset is offered, rendered graph states are in /results/test.png Training tensorboard output and results are stored in /results/results_Phase3simp/
Replicate the Scale-up experiment with the DQN-GNN model (section '6.2 Scale-up to real-world road networks')
```
python Phase2b_gnn-dqn.py --train_on NWB_AMS --max_nodes 975 --qnet s2v --train True --demoruns True
```
Note: to train with AMS graphs, 20GB of GPU VRAM is required. If this is not available, training can be performed on smaller graphs, e.g. 'M3M5Mix'. Training tensorboard output and results are stored in /results/results_Phase2/
Replicate the baseline PPO-GNN-LSTM experiment (section '6.3 Extend to partial observability')
```
python Phase3_lstm-gnn-ppo_simp.py --train_on NWB_AMS_mixed_obs --obs_mask mix --train True --demoruns True  --lstm_type EMB --lstm_hdim 64
```
Note: to train with AMS graphs, 20GB of GPU VRAM is required. Training tensorboard output and results are stored in /results/results_Phase3simp/

Background and examples

Goal: predicting escape routes in a passive search scenario with partial observability
Demo: a Graph Neural Net based policy model, trained using PPO with invalid action masking, can generalize and be applied to unseen graphs
Demo: escape agent traverses from Dam Square (Amsterdam) to a target node, while avoiding pursuers that move to observation positions. Escape behavior is based on graph representation learning on smaller graphs, combined with reinforcement learning using PPO
Demo: performance of the Collision Risk Avoidance heuristic benchmark
Demo: performance of GNN-LSTM model, trained using PPO under partial observability

Citation

@mastersthesis{weerd2022spe_rl,
    author = {R. van der Weerd},
    institution = {University of Amsterdam, Graduate School of Informatics},
    pages = 57,
    school = {University of Amsterdam, Graduate School of Informatics, Master's program in artificial intelligence},
    title = {Predicting plausible escape routes using reinforcement learning and graph representations},
    year = 2022
}