Tidy Reinforcement Learning with Tensorflow

I am a researcher working on automation tasks using deep reinforcement learning. The paper and the reality were quite different, and there was a lot of difficulty during the automation task. I create this repository to help those who start a task using deep reinforcement learning. All of the code is in tensorflow and Python 3.

Objective

The code in this repo has a simple and pretty structure in which each algorithm achieves uniformity. It will be a huge help for you to understand the differences between the reinforcement learning algorithms.
Take advantage of the pseudo code folder. You should be able to see pseudo code and make all the algorithms into a consistent architecture.

List of Implemented Algorithms

We used open-gym cartpoles (for discrete tasks), mountain car (for continous tasks), and pendulum (for continous tasks). In the case of HER, we used the coin flipping environment cited in the paper.

Policy Gradient (PG) for cartpole (discrete task)
Advantage Actor-Critic (A2C) for cartpole (discrete task)
Advantage Actor-Critic (A2C) for mountain car (continuous task),,,(imperfect...!!!)
Proximal Policy Optimization (PPO) (continuous task)
Deep Q Network (DQN) for cartpole (discrete task)
Deep Deterministic Policy Gradient (DDPG) for pendulum (continuous task)
Hindsight Experience Replay for coin flipping (discrete task),,,(imperfect...!!!)
Soft Actor-Critic (SAC) for pendulum (continuous task)

Papers / Pseudo Codes of RL Algorithms

Policy Gradient (PG) [Paper] [Pseudo Code]
Proximal Policy Optimization (PPO) [Paper] [Pseudo Code]
Asynchronous Advantage Actor-Critic (A3C) [Paper] [Pseudo Code]
Deep Q-learning Network (DQN) [Paper] [Pseudo Code]
Deep Deterministic Policy Gradient (DDPG) [Paper] [Pseudo Code]
Hindsight Experience Replay (HER) [Paper] [Pseudo Code]
Soft Actor-Critic (SAC) [Paper] [Pseudo Code]

Compare the Following Algorithms

PG (cartpole) vs A2C (cartpole)
A2C (cartpole) vs A2C (pendulum)
A2C (pendulum) vs PPO (pendulum)
PG (cartpole) vs DQN (cartpole)
A2C (pendulum) vs DDPG (pendulum)
DQN (cartpole) vs HER (coin)
PPO (pendulum) vs SAC (pendulum)

Some Tips in Realistic Development

A positive reward is a magnet, and a negative reward is a mole game.
The fastest agent to learn the Pendulum task is DDPG. So is ddpg the best reinforcement learning algorithm?
Which of the sparse reward and dense reward is practical?
If the problem is difficult, split it up and approach it.
Without a simulator, there is no answer other than model-based running.
In very difficulut task, it does not affect the skill drawn from experience replay.
Hindsight Experience Replay changes reward function. Unless you are a master of reinforcement learning, do not try HER.

Installation

pip install -r requirements.txt

voyagerahn/tidy-rl