/XWorld

A C++/Python simulator package for reinforcement learning

Primary LanguageC++Apache License 2.0Apache-2.0

XWorld

This repository contains a collection of simulators for Reinforcement Learning research.

Difficulty Name Description Thread-compatible?* Optional? Policy Teacher?
Easy SimpleGame A simple 1D array-walking game. Yes No Discrete No
Easy, Medium SimpleRace A simple synthetic car racing game. Yes No Discrete No
Medium, Hard Atari Wrappers for the Arcade Learning Environment (ALE) environment. For stability, we use a fork version. Yes Yes Discrete No
Medium, Hard XWorld2D A 2D world for an agent to learn vision and language abilities. No No Discrete
Continuous
Yes
Hard XWorld3D A 3D world for an agent to learn vision and language abilities. No Yes Discrete
Continuous
Yes

(*If yes, then multithreading can be used; otherwise multiprocessing is needed.)

Architecture

XWorld features a teacher infrastructure implemented as a scheduler of multiple Finite State Machines (FSMs). The idea is that given the environment, the teacher can propose a task sampled (by some heuristics) from a task set. Each task - formulated as an FSM - has several stages, and the teacher does different things in different stages. The transition from one stage to another is determined by the envionment state, e.g., whehter the agent is idle or whether it has achieved the goal. Each stage returns several things including the next stage and the teacher's action. Currently, we define language (strings) as the teacher's sole action. However, the teacher is able to change the environment (e.g., adding/deleting objects, changing the map size, etc.) within each stage.

The above figure illustrates the architecture. The motivation is to let the users flexibly write simple Python scripts to configure the environment maps and tasks.

Currently, the teacher is only incorporated into XWorld2D and XWorld3D.

Requirements

  • Compiler: GCC 4.8 or above
  • CMake: CMake 3.0 or above
  • Python: Python 2.7

Dependencies

The following softwares must be installed before building XWorld.

Boost, Glog, GFlags, GTest, and Python

In Ubuntu 14.04 and 16.04, you can do

sudo apt-get install libboost-all-dev libgflags-dev libgoogle-glog-dev libgtest-dev python-dev

Build

First get this git repository

git clone https://github.com/PaddlePaddle/XWorld

Suppose the directory is xworld_path, then do

cd <xworld_path>
mkdir -p build
cd build
cmake [<optional parameters>] ..

For example,

cd ~/XWorld; mkdir build; cd build
cmake ..

Finally, in the build directory do

make
make test

By default, XWorld only builds the first three games: SimpleGame, SimpleRace, and XWorld2D.

Optionally, you can install Atari by:

cmake -DWITH_ATARI=ON ..

which will automatically download and build Atari.

You can also install XWorld3D by:

cmake -DWITH_XWORLD3D=ON ..

Usage

Python interface

We provide a set of simple Python APIs for interacting with the simulators. After building XWorld, you need to export the path of the python module:

export PYTHONPATH=<xworld_path>/python:$PYTHONPATH

You can add the above line to ~/.bashrc to avoid doing the export in the future.

To get started, several examples of the simulator Python APIs can be found in

<xworld_path>/python/examples

C++ interface

Alternatively, several C++ examples (run the .sh scripts inside) can be found in

<xworld_path>/examples

These examples use the individual class constructors to create games. However, we also provide a unified simulator interface for creating games in a more convenient way (like in Python). A demo of the unified C++ simulator interface for multi-process simulation can be found in

<xworld_path>/examples/demo_interface.cpp

Generally, C++ APIs are more flexible but expose more details compared to the Python APIs.

Flags of a game

Option flags are passed into a game via different ways for the two interfaces:

  • Python: you need to provide the flags when creating the game as the arguments.
  • C++: you need to set the flags via GFlags in the command line.

For descriptions of the flags of a game, please take a look at the README file under the game directory.

Citations

If you use XWorld2D for research, consider citing

If you use XWorld3D for research, consider citing

If you use our wrappers of the third-party simulators, please follow their original guide for citation.

License

This repository has the Apache2.0 license, except that the third-party simulator ALE has its own license.