leihu0724/Deep-Reinforcement-Learning-Algorithms-with-Pytorch

Clean, Robust, and Unified PyTorch implementation of popular DRL Algorithms (Q-learning, Duel DDQN, PER, C51, Noisy DQN, PPO, DDPG, TD3, SAC, ASL)

Clean, Robust, and Unified PyTorch implementation of popular DRL Algorithms

1.Dependencies

This repository uses the following python dependencies unless explicitly stated:

gymnasium==0.29.1
numpy==1.26.1
pytorch==2.1.0

python==3.11.5

2.How to use my code

Enter the folder of the algorithm that you want to use, and run the main.py to train from scratch:

python main.py

For more details, please check the README.md file in the corresponding algorithm folder.

3. Separate links of the code

• 1.Q-learning
• 2.1Duel Double DQN
• 2.2Noisy Duel DDQN on Atari Game
• 2.3Prioritized Experience Replay(PER) DQN/DDQN
• 2.4Categorical DQN (C51)
• 2.5NoisyNet DQN
• 3.1Proximal Policy Optimization(PPO) for Discrete Action Space
• 3.2Proximal Policy Optimization(PPO) for Continuous Action Space
• 4.1Deep Deternimistic Policy Gradient(DDPG)
• 4.2Twin Delayed Deep Deterministic Policy Gradient(TD3)
• 5.1Soft Actor Critic(SAC) for Discrete Action Space
• 5.2Soft Actor Critic(SAC) for Continuous Action Space
• 6.Actor-Sharer-Learner(ASL)

4. Recommended Resources for DRL

4.1 Simulation Environments:

• gym and gymnasium (Lightweight & Standard Env for DRL; Easy to start; Slow):

• Isaac Gym (NVIDIA’s physics simulation environment; GPU accelerated; Superfast):

• Sparrow (Light Weight Simulator for Mobile Robot; DRL friendly):

• ROS (Popular & Comprehensive physical simulator for robots; Heavy and Slow):

• Webots (Popular physical simulator for robots; Faster than ROS; Less realistic):

• Envpool (Fast Vectorized Env)
• Other Popular Envs

4.2 Books：

• 《Reinforcement learning: An introduction》--Richard S. Sutton
• 《深度学习入门：基于Python的理论与实现》--斋藤康毅

4.3 Online Courses:

• RL Courses(bilibili)--李宏毅(Hongyi Li)
• RL Courses(Youtube)--李宏毅(Hongyi Li)
• UCL Course on RL--David Silver
• 动手强化学习--上海交通大学

4.4 Blogs:

• OpenAI Spinning Up
• Policy Gradient Theorem --Cangxi
• Policy Gradient Algorithms --Lilian
• Theorem of PPO
• The 37 Implementation Details of Proximal Policy Optimization
• Prioritized Experience Replay
• Soft Actor Critic
• A (Long) Peek into Reinforcement Learning --Lilian
• Introduction to TD3

5. Important Papers

DQN: Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. nature, 2015, 518(7540): 529-533.

Double DQN: Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning[C]//Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1).

Duel DQN: Wang, Ziyu, et al. "Dueling network architectures for deep reinforcement learning." International conference on machine learning. PMLR, 2016.

PER: Schaul T, Quan J, Antonoglou I, et al. Prioritized experience replay[J]. arXiv preprint arXiv:1511.05952, 2015.

C51: Bellemare M G, Dabney W, Munos R. A distributional perspective on reinforcement learning[C]//International conference on machine learning. PMLR, 2017: 449-458.

NoisyNet DQN: Fortunato M, Azar M G, Piot B, et al. Noisy networks for exploration[J]. arXiv preprint arXiv:1706.10295, 2017.

PPO: Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J]. arXiv preprint arXiv:1707.06347, 2017.

DDPG: Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[J]. arXiv preprint arXiv:1509.02971, 2015.

TD3: Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//International conference on machine learning. PMLR, 2018: 1587-1596.

SAC: Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//International conference on machine learning. PMLR, 2018: 1861-1870.

ASL: Train a Real-world Local Path Planner in One Hour via Partially Decoupled Reinforcement Learning and Vectorized Diversity

6. Training Curves of my Code:

Q-learning:

Duel Double DQN:

CartPole	LunarLander

Noisy Duel DDQN on Atari Game:

Pong	Enduro

Prioritized DQN/DDQN:

CartPole	LunarLander

Categorical DQN:

CartPole	LunarLander

NoisyNet DQN:

CartPole	LunarLander

PPO Discrete:

PPO Continuous:

DDPG:

Pendulum	LunarLanderContinuous

TD3:

SAC Continuous:

SAC Discrete:

Actor-Sharer-Learner (ASL):