Autonomous Racing Robot With an Arduino, a Raspberry Pi and a Pi Camera

Autonomous toy racing car. CAMaleon team at the Toulouse Robot Race 2017. Humbavision team at IronCar. Medium article: https://medium.com/@araffin/autonomous-racing-robot-with-an-arduino-a-raspberry-pi-and-a-pi-camera-3e72819e1e63

Video of the car: https://www.youtube.com/watch?v=xhI71ZdSh6k

Detailed Presentation
3D Models and Training Data
- 3D Models
- Training Data
  - IronCar and Toulouse Robot Race Datasets
How to run everything ?
Autonomous mode
Remote Control Mode
How to train the line detector ?
Installation
Contributors

Detailed Presentation

We wrote an article on medium that detailed our approach. You can read it here

En français: http://enstar.ensta-paristech.fr/blog/public/racing_car/

3D Models and Training Data

3D Models

3D models (we used onshape.com):

Note: the Battery Holder was designed for this External Battery

Training Data

IronCar and Toulouse Robot Race Datasets

We release the different videos taken with the on-board camera, along we the labeled data (the labels are in a pickle file) for IronCar and Toulouse Robot Race:

Videos
(outdated) Toulouse Dataset
(outdated) IronCar Dataset

How to run everything ?

For installation, see section Installation.

Autonomous mode

Compile and upload the code on the Arduino

cd arduino/
make
make upload

Launch the main script on the Raspberry Pi, it will try to follow a line. All useful constants can be found in constants.py.

python main.py

Remote Control Mode

You need a computer in addition to the raspberry pi
Create a Local Wifi Network (e.g. using create ap)
Connect the raspberry pi to this network (Wifi on RPI)
Launch teleoperation server (it will use the port 5556)

python -m teleop.teleop_server

Launch teleoperation client on your computer (you have to edit the raspberry pi IP in constants.py)

python -m teleop.teleop_client

Enjoy! You can now control the car with the keyboard.

How to train the line detector ?

Record a video in the teleoperation mode:

python -m teleop.teleop_server -v my_video

Convert the recorded video from h264 to mp4 using ffmpeg or MP4Box

MP4Box -add video.h264 video.mp4

Split the video into a sequence of images

python -m train.split_video -i video.mp4 -o path/to/dataset/folder

Label the data using the labeling tool: https://github.com/araffin/graph-annotation-tool
Rename the json file that contains the labels to labels.json and put it in the same folder of the dataset (folder with the images)
Train the neural network (again please change the paths in the script)

python -m train.train -f path/to/dataset/folder

The best model (lowest error on the validation data) will be saved as cnn_model_tmp.pth.

Test the trained neural network (you can use -i option to test it on a video)

python -m train.test -f path/to/dataset/folder -w cnn_model_tmp.pth

Installation

Recommended : Use an image with everything already installed

You need a 16GB micro sd card (warning, all data on that card will be overwritten) WARNING: for a smaller sd card, you need to resize the image before writing it (this link and repo may help)
Download the image here

Infos about the linux image: OS: Ubuntu MATE 16.04 for raspberry pi

Username: enstar

Password: enstar

Installed softwares:

all the dependencies for that project (OpenCV >= 3, PyTorch, ...)
the current project (in the folder RacingRobot/)
ROS Kinetic

Camera and SSH are enabled.

Identify the name of your sd card using:

fdisk -l

For instance, it gives:

/dev/mmcblk0p1            2048   131071   129024   63M  c W95 FAT32 (LBA)
/dev/mmcblk0p2          131072 30449663 30318592 14,5G 83 Linux

In that case, your sd card is named /dev/mmcblk0 (p1 and p2 stand for partition).

Write the downloaded image on the sd card.

gunzip --stdout ubuntu_ros_racing_robot.img.gz | sudo dd bs=4M of=/dev/mmcblk0

Enjoy! The current project is located in RacingRobot/.

If you want to back up an image of a raspberry pi:

sudo dd bs=4M if=/dev/mmcblk0 | gzip > ubuntu_ros_racing_robot.img.gz

From Scratch

Update your pi

sudo apt-get update
sudo apt-get upgrade
sudo rpi-update

Arduino + Arduino Makefile + rlwrap + screen + MP4Box

sudo apt-get install arduino-core arduino-mk rlwrap screen gpac

Arduino 1.0.5
Arduino-Makefile
OpenCV >= 3
libserial-dev (apt-get)
Python 2 or 3

OpenCV

PreCompiled This is the recommended method Compile from source:
Docs
Tutorial
Raspbian Tuto

Python Packages

All the required packages (except pytorch and torchvision) can be found in requirements.txt, install them using:

pip install -r requirements.txt

In short:

PySerial
TQDM (for progressbar)
PyGame (for teleoperation)
Enum support (for Python 2)
ZeroMQ (for teleoperation)
Pytorch (you have to compile it from source for the RPI)
scikit-learn
scipy

pip install pyserial tqdm pygame enum34 scikit-learn scipy

ZeroMQ (Message Passing with sockets) for remote control mode

sudo apt-get install libzmq3-dev
pip install pyzmq

Note: for using the serial port, you need to change current user permissions:

# Add user to dialout group to have the right to write on the serial port
sudo usermod -a -G dialout $USER
# You need to logout/login again for that change to be taken into account

Additional python dev-dependencies for training the neural network: On your laptop:

pip install pytorch
pip install torchvision

On the raspberry pi :

You can try to use Python 2 wheel (not tested) that was created for this project:

Download Python Wheel here

And then:

pip install torch-0.4.0a0+b23fa21-cp27-cp27mu-linux_armv7l.whl

Or follow this tutorial: PyTorch on the Raspberry Pi

Make sure you have at least 3 Go of Swap. (see link above)

sudo dd if=/dev/zero of=/swap1 bs=1M count=3072 Status=progress
sudo mkswap /swap1
sudo swapon /swap1

(optional) Install a recent version of cmake + scikit-build + ninja
Install PyTorch

See https://github.com/pytorch/pytorch for dependencies. Additional dependencies:

sudo apt-get install libopenblas-dev libeigen3-dev libffi-dev

# don't forget to set the env variables:
export NO_CUDA=1
export NO_DISTRIBUTED=1
git clone --recursive https://github.com/pytorch/pytorch
python setup.py install --user
# torchvision is not used yet
pip install torchvision --user

Wifi on RPI

OpenCV with Python 2 (RPI), compiling from source (cf Docs and tuto): Additional dependencies:

sudo apt-get install libtbb-dev opencl-headers libomp-dev libopenblas-dev libeigen3-dev
sudo apt-get install libatlas-base-dev gfortran -y

with Gstreamer Then:

cmake -D CMAKE_INSTALL_PREFIX=/usr/local \
-D BUILD_opencv_java=OFF \
-D BUILD_opencv_python2=ON \
-D BUILD_opencv_python3=OFF \
-D PYTHON_DEFAULT_EXECUTABLE=$(which python) \
-D INSTALL_C_EXAMPLES=OFF \
-DINSTALL_PYTHON_EXAMPLES=ON -DBUILD_TIFF=ON -DWITH_CUDA=OFF -DWITH_OPENGL=ON -DWITH_OPENCL=ON -DWITH_IPP=ON -DWITH_TBB=ON -DWITH_EIGEN=ON -DWITH_V4L=ON -DWITH_VTK=OFF -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DCMAKE_BUILD_TYPE=RELEASE ..

OpenCV with Anaconda, compiling from source:

cmake -DPYTHON_EXECUTABLE=/home/ỳour_name/anaconda3/bin/python3 \
-DPYTHON_INCLUDE=/home/ỳour_name/anaconda3/include \
-DPYTHON_LIBRARY=/home/ỳour_name/anaconda3/lib/libpython3.6m.so \
-DPYTHON_PACKAGES_PATH=/home/ỳour_name/anaconda3/lib/python3.6/site-packages \
-DPYTHON_NUMPY_INCLUDE_DIR=/home/ỳour_name/anaconda3/lib/python3.6/site-packages/core/include -DINSTALL_PYTHON_EXAMPLES=ON -DBUILD_TIFF=ON -DBUILD_opencv_java=OFF -DWITH_CUDA=OFF -DWITH_OPENGL=ON -DWITH_OPENCL=ON -DWITH_IPP=ON -DWITH_TBB=ON -DWITH_EIGEN=ON -DWITH_V4L=ON -DWITH_VTK=OFF -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DCMAKE_BUILD_TYPE=RELEASE ..

Docker Support (Experimental)

Info: http://www.hotblackrobotics.com/en/blog/2018/01/22/docker-images-arm/

Build docker image (laptop image):

docker build . -f docker/Dockerfile.cpu -t racing-robot-cpu

Build docker image (raspberry pi image):

docker build . -f docker/DockerfileBase.rpi3 -t racing-robot-rpi3-base

docker build . -f docker/Dockerfile.rpi3 -t racing-robot-rpi3

Contributors

Sergio Nicolas Rodriguez Rodriguez
Antonin Raffin

ReedMark/RacingRobot