The goal of this project is to create an agent to play a game in a virtual world. An agent here can be seen as a player, but the player is the computer which use artificial intelligence. The virtual world are full of two types of bananas: the fresh one(yellow) and the rotten one(blue). The agent need to collect the yellow bananas as many as possible and avoid the blue one.
The virtual world can be seen in the following .gif file.
In order to successfully create a smart agent, we use Deep Reinforcement Learning(DRL) algorithm which is one well-known method in Artificial Intelligent. DRL algorithm allow the agent, in our case the player, learn how to play the while interacting in the virtual world. In other words, the agent starts the game without any knowledge, learns slowly by interacting the game and finally masters the game after enough interaction in the virtual world.
Moreover, we use the Deep Q-Networks(DQN) inspired by the paper which described successful implementation of Reinforcement Learning algorithm to play in various Atari games.
You are encouraged to learn the concept of Reinforcement Learning and the DQN paper and use this repo to better understand the implementation of an DQN agent to play the banana game.
In reinforcement learning, the virtual world where the agent(the player) play is usually called the environment. Thus, the word environment refers to the square world full of bananas in our case.
The environment is a virtual world full of bananas explained in the introduction section. The environment is further processed to a simple representation of state space. The state space has 37 dimensions and contains the agent's velocity, along with ray-based perception of objects around agent's forward direction
The agent is the player here. The agent will get a reward of +1 for collecting a yellow banana, and a reward of -1 is provided for collecting a blue banana. Thus, the goal of our agent is to collect as many yellow bananas as possible while avoiding blue bananas.
Given this information, the agent has to learn how to best select actions. Four discrete actions are available, corresponding to:
0- move forward.1- move backward.2- turn left.3- turn right.
Follow the instructions below to explore the environment on your own machine! You will also learn how to use the Python API to control your agent.
To set up your python environment to run the code in this repository, follow the instructions below.
-
Create (and activate) a new environment with Python 3.6.
- Linux or Mac:
conda create --name drlnd python=3.6 source activate drlnd- Windows:
conda create --name drlnd python=3.6 activate drlnd
-
Follow the instructions in this repository to perform a minimal install of OpenAI gym.
-
Clone the repository (if you haven't already!), and navigate to the
python/folder. Then, install several dependencies.
git clone https://github.com/jhonsonlee/dqn-navigating-in-a-virtual-world.git
cd dqn-navigating-in-a-virtual-world/python
pip install .- Create an IPython kernel for the
drlndenvironment.
python -m ipykernel install --user --name drlnd --display-name "drlnd"- Before running code in a notebook, change the kernel to match the
drlndenvironment by using the drop-downKernelmenu.
(For Windows users) The ML-Agents toolkit supports Windows 10. While it might be possible to run the ML-Agents toolkit using other versions of Windows, it has not been tested on other versions. Furthermore, the ML-Agents toolkit has not been tested on a Windows VM such as Bootcamp or Parallels.
-
Download the environment from one of the links below. You need only select the environment that matches your operating system:
- Linux: click here
- Mac OSX: click here
- Windows (32-bit): click here
- Windows (64-bit): click here
(For Windows users) Check out this link if you need help with determining if your computer is running a 32-bit version or 64-bit version of the Windows operating system.
(For AWS) If you'd like to train the agent on AWS (and have not enabled a virtual screen), then please use this link to obtain the "headless" version of the environment. You will not be able to watch the agent without enabling a virtual screen, but you will be able to train the agent. (To watch the agent, you should follow the instructions to enable a virtual screen, and then download the environment for the Linux operating system above.)
-
Place the file in the DRLND GitHub repository, in the
p1_navigation/folder, and unzip (or decompress) the file.
The code in this repository will be written mostly in .ipynb file. Thus, you need to get familiar with the jupyter notebook which is a great editor today. Please, take your time to know Jupyter Notebook in this blogpost.
There are three important files which contains the source code to run and train our agent:
Navigation.ipynbcontains the instructions to explore the environent, train the agent to be smart and run the smart agent./src/agent.pycontains the source code which describes how the agent works/src/model.pycontains the source code of deep learning model for the DQN value function
In order to understand how to create a smart agent, we must recognize the environment first. Please, open Navigation.ipynb using Jupyter Notebook and follow these steps:
- Step 1 : Start the Environment
- Step 2 : Examine the State and Action Spaces
- Step 3 : Take Random Actions in the Environment
As you can see in the Explore the Environment section, the agent is still taking random actions and generate zero points during the gameplay. In order to create a smart agent, we need to train the agent by following these two steps:
- Step 4 : Contruct the Train Method
- Step 5 : Train the Agent
After we train to agent how to better play the game, we can watch the agent play with better decision and better score. Howevever, we can not see the agent play in live gameplay in this notebook. But, we can see the agent collects yellow bananas and avoids blue bananas by inspecting the score movement. Please follow the Step 6 : Watch the Agent Play.
This repository is under the MIT license.