Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning

Jie Xu, Tao Du, Michael Foshey, Beichen Li, Bo Zhu, Adriana Schulz, Wojciech Matusik

ACM Transactions on Graphics, 38(4) 42:1-42:12 (SIGGRAPH), 2019

This repository implements the code for the paper Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning (SIGGRAPH 2019).

Code Structure

SimulationUI: C++ rigid body simulation to visualize the controller's performance. (C++)
Training: Code to train the controller with our method with Reinforcement Learning. (Python)
Firmware: Modified Ardupilot firmware code to implement our controller on real hardware. (C++)

Simulation UI

The code is implemented to visualize the controller's performance in a realistic simulation environment and provide the user interaction to control the hybrid UAV through keyboard.
The code has been tested on Ubuntu 16.04 and Ubuntu 18.04.
Compiling the code relies on cmake >= 3.1.0. Please install it through the instruction in this link before compiling the code.
Compile the code:
- Enter SimulationUI folder.
- Create build folder and enter it by:
```
mkdir build; cd build;
```
- Run cmake and make to compile the code.
```
cmake -DCMAKE_BUILD_TYPE=Release ../
make -j4
```
- If cmake fails, please install the missing libraries listed in the error message.
Run the code:
- The default code will run the simulation for QuadPlane controlled by a pre-trained neural network controller.
- Run the following command inside the build folder:
```
./projects/design_viewer/design_viewer
```
- It will open a window as below where you can control the hybrid UAV through keyboard commands.
Keyboard Command:
- We provide a bunch of keyboard command to allow you to easily control the hybrid UAV.
- p: start/pause simulation, you need to press p after you create the window.
- w/s: increase/decrease velocity z (vertical velocity, negative value is going up).
- a/d: increase/decrease target yaw angle.
- up/down: increase/decrease velocity x (forward velocity).
- left/right: increase/decrease velocity y (side velocity).
- You can use the mouse scroll wheel to zoom in/out (scroll) and change the view (press and move) of the camera.
- For more keyboard operation, please refer to the code.
- To be noted, there is a dead zone for the velocity z, thus the controller is not triggered until the velocity z is smaller than -0.5.
Tail-sitter: we also provide the config file and the pretrained controller for tail-sitter. You can modify the code in DesignViewer.cpp to load its config file and replace the controller parameter file projects/copter_simulation/Controller/NN/NNModelParameters.h with its controller file (which you can find in projects/copter_simulation/Controller/NN/TailSitter/).

Training

The code has been tested on Ubuntu 16.04.
The code relies on a modified version of OpenAI baselines in git@github.com:eanswer/openai_baselines.git. To run the code, please first clone the repository
```
git clone git@github.com:eanswer/openai_baselines.git
```
and then follow the README.md to install baselines.
We provide two models in the data folder.
To start training a controller for a model, for example QuadPlane, in one thread,
```
python run.py --model QuadPlane
```
or in multiple threads (e.g. 4),
```
mpirun -np 4 python run.py --model QuadPlane
```
You can also add a --visualize to visualize the training by Mujoco. (If you don't install Mujoco, this option does't work).
The trained models are saved in results folder.

To test a trained mode, for example model_afterIter_400.ckpt, you can run

python run.py --model QuadPlane --controller model_afterIter_400 --play

To save a trained model to a c++ format to be plugged into the SimulationUI code or ardupilot firmware code.
```
python save.py --controller model_afterIter_400
```

Firmware Deployment

The firmware code is based on the Ardupilot codebase and tested in Pixhawk 1.0.
First plug the model description file generated by training code (should named AP_MotorsPolicyDefinition.h) into the firmware codebase:
```
cp AP_MotorsPolicyDefinition.h ../../Firmware/ardupilot/libraries/AP_Motors/
```
Connect the Pixhawk 1.0 to your computer by usb cabel.
Enter the folder ArduCopter by cd Firmware/ardupilot/ArduCopter.
Compile and upload the firmware code by
```
make px4-v2-upload
```
Please refer to Ardupilot official document for more detials.

Citation

If you find our paper or code is useful, please consider citing: