This is the code for our paper: Tongtong Liu,Joe McCalmon, Cameron Lischke, Asif Rahman, Talal Halabi, and Sarra Alqahtani, On Weaponizing Actions in Multi-Agent Reinforcement Learning: Theoretical and Empirical Study on Security and Robustness, The 24th International Conference on Principles and Practice of Multi-Agent Systems, 2022, accepted
Note: Environments include requirements.txt file in each folder.
In order to install virtual environments:
cd
into the corresponding directories- type
pip install -r requirements.txt
This project utilized the following GitHub as referenced, and if encountered any problem regarding installment of the environment, user should follow the corresponding GitHub in the following:
MADDPG: https://github.com/openai/maddpg
Multi-agent-particle: https://github.com/openai/multiagent-particle-envs
DDPG/PPO: https://github.com/MorvanZhou/Reinforcement-learning-with-tensorflow
AIRL: https://github.com/yangmuzhi/airl
Qmix on STARCRAFT II: https://github.com/deepmind/pysc2
Python version should be 3.6
User should also add multi-agent-particle folder to PYTHONPATH for MADDPG to run successfully
./Attacks for MADDPG/
contains code for all attacks in cooeprative navigation and physical deception./Attacks for Qmix/
contains code for all attacks in StarCraft II./MARL_Detection/
contains code for the detection model and FGSM attack./Demo Video/
contains the visualization of the attack in different environments
- code are available in
./experiments/predicted_agent_0_attacks.py
- Whitebox, blackbox, random and strategically-timed attacks can be performed for cooperative navigation and physical deception from the 'attack' function in predicted_agent_0_attacks.py
- Parameters threshold, attack_rate, random, and black_box control which attack is run.
- threshold is the threshold of c(s_t) for the strategically timed attack
- To run cooperative navigation, set the flag --scenario=simple_spread. For physical deception, set --scenario=simple_adversary
- packages and dependencies are available in requirements.txt
- all code for attacks are in the folder
./experiments/
- CN: stands for cooperative navigation
- PD: stands for physical deception (simple adversary)
- KL: stands for counterfactual attack (simple spread)
- Adv: stands for zero-sum attack
- all trained weights are availble in corresponding folders
- For example: if you want to run counterfactual, blackbox, in cooperative navigation, you should type
python train_PPO_KL_CN_Blackbox.py --scenario simple_spread
- attack rate can be specified by variable
attack_rate
, range from 0 to 1 - For PPO, please go back to
PPO.py
to change the state dimension - 23 for CN (18 + 5)
- 15 for PD (15 + 5)
- to install the StarCraft II game, please refer back to StarCraft II's original GitHub for installment procedure in different operating system
- to install the dependency, please check the requirements.txt
- all implementations are in
rollout.py
under functiongenerate_episode()
- attack name can be changed in
main.py
using variable args.attack_name: 'inverse_glob' for zero sum
'counterfactual_RL' for counterfactual
'strategic' for strategic
'random' for random
- attack rate can be specified in
rollout.py
using variable nameattack_rate
- white box/black box can be specified in arguments.py under white_box (True for WB, False for BB)
- other important output info can be found in
runner.py
under functionevaluate()
- please follow the "Anomalous behavior detection in c-MARL.pdf" in the folder
- Please see Appendix.pdf for more supplmentary details for this paper.