This repository contains deep reinforcement learning agents implemented in PyTorch to play the classic Snake game. The agents use DQN, DDQN, and A2C algorithms to learn policies to maximize the snake's length by collecting apples while avoiding collisions.
The animation above shows the Snake game environment. The snake needs to collect apples to increase its length while avoiding the walls and its own body.
Three different deep RL algorithms were implemented and compared:
DQN: Deep Q-Network (Q-Learning)
DDQN: Double Deep Q-Network (Q-Learning)
A2C: Advantage Actor-Critic
DQN and DDQN use deep neural networks to approximate the Q-value function. A2C uses separate actor and critic networks.
Each of the algorithms have a distinct jupyter notebook. To run the code, you will need Python 3 nand run the following libraries installed:
- PyTorch
- Gym
- NumPy
- Matplotlib
Simply open the notebook and run all cells to train each agent from scratch and evaluate them.
![Test-Results](https://private-user-images.githubusercontent.com/46433931/271803347-cbb8ef9e-d10d-4457-bdb0-0c6e8819d8df.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MTAzNzksIm5iZiI6MTcyNDQxMDA3OSwicGF0aCI6Ii80NjQzMzkzMS8yNzE4MDMzNDctY2JiOGVmOWUtZDEwZC00NDU3LWJkYjAtMGM2ZTg4MTlkOGRmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDEwNDc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRkMjU2NThhZWZmZWFmNDhiMmFjZDU0YTcwNzYwODMzNTNhMzNlNjJkNTFmYWFiMjQwNzk0NGZkM2I2NmRjMGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ZfT19mDGXkNUppeuanoc1j9eh0-eStKfbPCaAMuepAs)
The agents were trained for over 100,000 episodes. DDQN achieved the best performance, reaching 10 apples per episode during evaluation. DQN reached 7 apples, while A2C only managed 1 apple per episode.
DQN:
![DQN](https://private-user-images.githubusercontent.com/46433931/271803361-8ed26037-0cf4-4bec-92fc-e4e5fd62536c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MTAzNzksIm5iZiI6MTcyNDQxMDA3OSwicGF0aCI6Ii80NjQzMzkzMS8yNzE4MDMzNjEtOGVkMjYwMzctMGNmNC00YmVjLTkyZmMtZTRlNWZkNjI1MzZjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDEwNDc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTUzY2E0ZjkxYjNiNzU0ODA2YzliMWU3NDlhYzIxNGEyNzA1YjlmNjI0OWNiZWI2ZjU3NzY5Nzc1MjE4MDlhODUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.FMt6a1Q7r1sAgiGCla5zWQbXB89O57D9Md74m50D8hQ)
Training result: 12 apples per episode, Test result: 10 apples per episode
DDQN:
![ddqn](https://private-user-images.githubusercontent.com/46433931/271803369-bcb54350-d0b5-481a-89bd-1b1b8849a5c5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MTAzNzksIm5iZiI6MTcyNDQxMDA3OSwicGF0aCI6Ii80NjQzMzkzMS8yNzE4MDMzNjktYmNiNTQzNTAtZDBiNS00ODFhLTg5YmQtMWIxYjg4NDlhNWM1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDEwNDc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQ1MjQxNDkwNGM3N2Q0MGE1Yjk1YzIzYWU0MDA3ZDAzNzFjNjM2M2ViNWNlNmUxODhkN2E1N2FjNTA3MTBjMDMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.X90gykMtvy0LlD1FV21rCfi8rQkPQlyPvtwnGWyobgY)
Training result: 10 apples per episode, Test result: 7 apples per episode
A2C:
![a2c](https://private-user-images.githubusercontent.com/46433931/271803374-e26393f7-7458-4e83-8533-23d98212cc9f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MTAzNzksIm5iZiI6MTcyNDQxMDA3OSwicGF0aCI6Ii80NjQzMzkzMS8yNzE4MDMzNzQtZTI2MzkzZjctNzQ1OC00ZTgzLTg1MzMtMjNkOTgyMTJjYzlmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDEwNDc1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM5YTg4ZTY0ZWFmNTI3MTAwYmFiNGMxNTVmZmY5ODRhYmM0YjdmNDIyMjE5ODBlZWViMjZlZmRiY2E3ZjU5NTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.z08W_-bqysyCNwrMoRmDd1VPUc6gZfWY9untcwvOjFM)
Training result: 1 apple per episode, Test result: 1 apple per episode