TensorForce: A TensorFlow library for applied reinforcement learning

Introduction

TensorForce is an open source reinforcement learning library focused on providing clear APIs, readability and modularisation to deploy reinforcement learning solutions both in research and practice. TensorForce is built on top of TensorFlow and compatible with Python 2.7 and ≥3.5 and supports multiple state inputs and multi-dimensional actions to be compatible with Gym, Universe, and DeepMind lab. It further provides an easily extensible interface to implement new environments.

Finally, TensorForce aims to move all reinforcement learning logic into the TensorFlow graph, including control flow. This both reduces dependencies on the host language (Python), thus enabling portable computation graphs that can be used in other languages and contexts, and improves performance.

More information on architecture can also be found on our blog. Please also read the TensorForce FAQ if you encounter problems or have questions. Finally, you can sign up to our Gitter channel.

Finally, read the latest update notes (UPDATE_NOTES.md) for an idea of how the project is evolving, especially concerning majorAPI breaking updates.

The main difference to existing libraries is a strict separation of environments, agents and update logic that facilitates usage in non-simulation environments. Further, research code often relies on fixed network architectures that have been used to tackle particular benchmarks. TensorForce is built with the idea that (almost) everything should be optionally configurable and in particular uses value function template configurations to be able to quickly experiment with new models. The goal of TensorForce is to provide a practitioner's reinforcement learning framework that integrates into modern software service architectures.

TensorForce is actively being maintained and developed both to continuously improve the existing code as well as to reflect new developments as they arise. The aim is not to include every new trick but to adopt methods as they prove themselves stable.

Features

TensorForce currently integrates with the OpenAI Gym API, OpenAI Universe, the Unreal Engine (game engine), DeepMind lab, ALE and Maze explorer. The following algorithms are available (all policy methods both continuous/discrete and using a Beta distribution for bounded actions).

A3C using distributed TensorFlow or a multithreaded runner - now as part of our generic Model usable with different agents. - paper
Trust Region Policy Optimization (TRPO) - trpo_agent - paper
Normalised Advantage functions (NAFs) - naf_agent - paper
DQN - dqn_agent - paper
Double-DQN - ddqn_agent - paper
N-step DQN - dqn_nstep_agent
Vanilla Policy Gradients (VPG/ REINFORCE) - vpg_agent- paper
Actor-critic models - via baseline for any policy gradient model (see next list) - paper
Deep Q-learning from Demonstration (DQFD) - paper
Proximal Policy Optimisation (PPO) - ppo_agent - paper
Random and constant agents for sanity checking: random_agent, constant_agent

Other heuristics and their respective config key that can be turned on where sensible:

Generalized advantage estimation - gae_lambda - paper
Prioritizied experience replay - memory type prioritized_replay - paper
Bounded continuous actions are mapped to Beta distributions instead of Gaussians - paper
Baseline / actor-critic modes: Based on raw states (states) or on network output (network). MLP (mlp), CNN (cnn) or custom network (custom). Special case for mode states: baseline per state + linear combination layer (via baseline=dict(state1=..., state2=..., etc)).
Generic pure TensorFlow optimizers, most models can be used with natural gradient and evolutionary optimizers
Preprocessing modes: normalize, standardize, grayscale, sequence, clip, divide, image_resize
Exploration modes: constant,linear_decay, epsilon_anneal, epsilon_decay, ornstein_uhlenbeck

Installation

We uploaded the latest stable version of TensorForce to PyPI. To install, just execute:

pip install tensorforce

If you want to use the latest version from GitHub, use:

git clone git@github.com:reinforceio/tensorforce.git
cd tensorforce
pip install -e .

TensorForce is built on Google's Tensorflow. The installation command assumes that you have tensorflow or tensorflow-gpu installed.

Alternatively, you can use the following commands to install the tensorflow dependency.

To install TensorForce with tensorflow (cpu), use:

# PyPI install
pip install tensorforce[tf]

# Local install
pip install -e .[tf]

To install TensorForce with tensorflow-gpu (gpu), use:

# PyPI install
pip install tensorforce[tf_gpu]

# Local install
pip install -e .[tf_gpu]

To update TensorForce, use pip install --upgrade tensorforce for the PyPI version, or run git pull in the tensorforce directory if you cloned the GitHub repository. Please note that we did not include OpenAI Gym/Universe/DeepMind lab in the default install script because not everyone will want to use these. Please install them as required, usually via pip.

Examples and documentation

For a quick start, you can run one of our example scripts using the provided configurations, e.g. to run the TRPO agent on CartPole, execute from the examples folder:

python examples/openai_gym.py CartPole-v0 -a examples/configs/ppo.json -n examples/configs/mlp2_network.json

Documentation is available at ReadTheDocs. We also have tests validating models on minimal environments which can be run from the main directory by executing pytest{.sourceCode}.

Create and use agents

To use TensorForce as a library without using the pre-defined simulation runners, simply install and import the library, then create an agent and use it as seen below (see documentation for all optional parameters):

from tensorforce.agents import PPOAgent

# Create a Proximal Policy Optimization agent
agent = PPOAgent(
    states_spec=dict(type='float', shape=(10,)),
    actions_spec=dict(type='int', num_actions=10),
    network_spec=[
        dict(type='dense', size=64),
        dict(type='dense', size=64)
    ],
    batch_size=1000,
    step_optimizer=dict(
        type='adam',
        learning_rate=1e-4
    )
)

# Get new data from somewhere, e.g. a client to a web app
client = MyClient('http://127.0.0.1', 8080)

# Poll new state from client
state = client.get_state()

# Get prediction from agent, execute
action = agent.act(state)
reward = client.execute(action)

# Add experience, agent automatically updates model according to batch size
agent.observe(reward=reward, terminal=False)

Benchmarks

We provide a seperate repository for benchmarking our algorithm implementations at reinforceio/tensorforce-benchmark.

Docker containers for benchmarking (CPU and GPU) are available.

This is a sample output for CartPole-v0, comparing VPG, TRPO and PPO:

Please refer to the tensorforce-benchmark repository for more information.

Use with DeepMind lab

Since DeepMind lab is only available as source code, a manual install via bazel is required. Further, due to the way bazel handles external dependencies, cloning TensorForce into lab is the most convenient way to run it using the bazel BUILD file we provide. To use lab, first download and install it according to instructions https://github.com/deepmind/lab/blob/master/docs/build.md:

git clone https://github.com/deepmind/lab.git

Add to the lab main BUILD file:

package(default_visibility = ["//visibility:public"])

Clone TensorForce into the lab directory, then run the TensorForce bazel runner. Note that using any specific configuration file currently requires changing the Tensorforce BUILD file to adjust environment parameters.

bazel run //tensorforce:lab_runner

Please note that we have not tried to reproduce any lab results yet, and these instructions just explain connectivity in case someone wants to get started there.

Community and contribution guidelines

TensorForce is developed by reinforce.io, a new project focused on providing reinforcement learning software infrastructure. For any questions, get in touch at contact@reinforce.io.

Please file bug reports and feature discussions as GitHub issues in first instance. Please read the FAQ before creating an issue.

Please appreciate that we do not have the resources to help you find the right configuration for your problem, so unless you are reasonably convinced there is a bug (e.g. by testing known hyper-parameters), please do not create issues such as 'Algorithm X is not working on environment Y with Configuration Z' without showing you have done some research (again, please read the FAQ on why).

There is also a developer chat you are welcome to join. For joining, we ask to provide some basic details how you are using TensorForce so we can learn more about applications and our community. Please fill in this short form which will take you to the chat after.

Cite

If you use TensorForce in your academic research, we would be grateful if you could cite it as follows:

@misc{schaarschmidt2017tensorforce,
    author = {Schaarschmidt, Michael and Kuhnle, Alexander and Fricke, Kai},
    title = {TensorForce: A TensorFlow library for applied reinforcement learning},
    howpublished={Web page},
    url = {https://github.com/reinforceio/tensorforce},
    year = {2017}
}

We are also very grateful for our open source contributors (listed according to github): Islandman93, wassname, lefnire, Mazecreator, trickmeyer, mryellow, ImpulseAdventure, vwxyzjn, beflix, tms1337, BorisSchaeling, ngoodger, ekerazha, Davidnet, nikoliazekter, AdamStelmaszczyk, 10nagachika, petrbel, Kismuz.

h3dema/tensorforce