NA²Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning

Code for NA²Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning accepted by ICML 2023. NA²Q is implemented in PyTorch and tested on challenging tasks LBF and SMAC as benchmarks, which is based on PyMARL. [paper] [code]

Python MARL framework

This PyMARL includes baselines of the following algorithms:

QMIX: QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
COMA: Counterfactual Multi-Agent Policy Gradients
VDN: Value-Decomposition Networks For Cooperative Multi-Agent Learning
IQL: Independent Q-Learning
QTRAN: QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
WQMIX: Weighted Qmix: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
Qatten: Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
QPLEX: Qplex: Duplex Dueling Multi-Agent Q-Learning
MASAC: Actor-Attention-Critic for Multi-Agent Reinforcement Learning
SHAQ: SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
SQDDPG: QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
UPDET: Updet: Universal Multi-Agent Reinforcement Learning via Policy Decoupling with Transformers
CDS: Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

Thanks to all the original authors!

Installation instructions

Build the Dockerfile using

cd docker
bash build.sh

Set up StarCraft II and SMAC:

bash install_sc2.sh

This will download SC2 into the 3rdparty folder and copy the maps necessary to run over.

Requirements

Python 3.6+
pip packages listed in requirements.txt

The requirements.txt file can be used to install the necessary packages into a virtual environment (not recommended).

Run an experiment

# demo sc2
python src/main.py --config=qnam --env-config=sc2 with env_args.map_name=8m_vs_9m gpu_id=0 t_max=2010000 epsilon_anneal_time=50000 seed=1
# demo foraging
nohup python src/main.py --config=qnam --env-config=foraging with env_args.map_name=lbf-4-2 use_cuda=False seed=1 &

The config files act as defaults for an algorithm or environment.

They are all located in src/config. --config refers to the config files in src/config/algs --env-config refers to the config files in src/config/envs

All results will be stored in the Results folder.

Citing NA²Q

If you find this repository useful for your research, please cite it in BibTeX format:

@inproceedings{liu2023na2q,
  title = 	 {{NA$^2$Q}: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning},
  author =       {Liu, Zichuan and Zhu, Yuanyang and Chen, Chunlin},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {22539--22558},
  year = 	 {2023},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  url = 	 {https://proceedings.mlr.press/v202/liu23be.html},
}

In case of any questions, bugs, suggestions or improvements, please feel free to drop me or open an issue.

If you need the experimental results data from our paper, please contact me at zichuanliu@smail.nju.edu.cn and I'll be happy to share them!

zichuan-liu/NA2Q

NA2Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning