This repository contains an implementation of the multiple agent version of the Deep Deterministic Policy Gradient (DDPG) algorithm described in Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. The implementation has been done in PyTorch, in the Unity environment called Tennis, where two agents try to pass the ball over the net.
See more information about the environment in the section Environment details.
The agent is implemented and trained in the notebook multi-agent-reinforcement-learning/muti-agent-tennis-grid-search-minimal.ipynb
.
To see a comprehensive results report, see the report.md
file in this repository.
The agent will interact with a version of the Tennis Unity environment.
A reward of +0.1 is provided to each agent for each time it makes the ball pass over the net. If an agent lets the ball hit the ground or get out of bounds, it will have a reward of -0.01.
The observation space consists of 8 variables corresponding to position and velocity of the raquet and the ball.
Each action is a vector with two numbers per agent, corresponding to moving horizontally or vertically.
In order to solve the environment, the agents must have an average score of +0.5 over 100 consecutive episodes.
To be able to run the notebooks, one needs to prepare the environment and download the Unity environment.
As described in the Udacity github repo, to set up your python environment, follow the instructions below.
-
Create (and activate) a new environment with Python 3.6.
- Linux or Mac:
conda create --name drlnd python=3.6 source activate drlnd
- Windows:
conda create --name drlnd python=3.6 activate drlnd
-
Follow the instructions in this repository to perform a minimal install of OpenAI gym.
-
Clone the repository (if you haven't already!), and navigate to the
python/
folder. Then, install several dependencies.
git clone https://github.com/udacity/deep-reinforcement-learning.git
cd deep-reinforcement-learning/python
pip install .
- Create an IPython kernel for the
drlnd
environment.
python -m ipykernel install --user --name drlnd --display-name "drlnd"
- Before running code in a notebook, change the kernel to match the
drlnd
environment by using the drop-downKernel
menu.
Select and download the environment that matches your operating system:
Linux: click here
Mac OSX: click here
Windows (32-bit): click here
Windows (64-bit): click here
Then, place the file in the p3_collab-compet/
folder in the DRLND GitHub repository, and unzip (or decompress) the file.
(For Windows users) Check out this link if you need help with determining if your computer is running a 32-bit version or 64-bit version of the Windows operating system.
(For AWS) If you'd like to train the agent on AWS (and have not enabled a virtual screen), then please use this link to obtain the "headless" version of the environment. You will not be able to watch the agent without enabling a virtual screen, but you will be able to train the agent. (To watch the agent, you should follow the instructions to enable a virtual screen, and then download the environment for the Linux operating system above.)
Once your environment is set-up, just run the notebook multi-agent-reinforcement-learning/muti-agent-tennis-grid-search-minimal.ipynb
.