Autonomous Driving Framework

Objective

To design and implement an end-to-end neural network for autonomous driving using simpler CNN framework.

Tools used

LGSVL simulator as driving simulator
ROS2 nodes to receive sensor data from LGSVL Simulator
Preprocessing and feature engineering using Python, OpenCV
Keras/Tensorflow deep learning CNN model to train the data
Containerised docker image to run simulation using the GPU

It is assumed that you have some experience in using Linux, Git, ROS, Docker, web browsers, and Python.

This project is the modified version of ROS2 End-to-End Lane Following Model with LGSVL Simulator which is in turn inspired by NVIDIA's End-to-End Deep Learning Model for Self-Driving Cars.

Complete End-to-End Architecture

At the highest-level, the architecture consists of four modules: sensor module, data collection, pre-processing and training module, and finally evaluation module.

Features

Training mode: Manually drive the vehicle and collect data
Autonomous Mode: The vehicle drives itself based on Lane Following model trained from the collected data
ROS2-based
- Time synchronous data collection node
- Deploying a trained model in a node
Data preprocessing for training
- Data normalization and manipulation
- Splitting data into training set and test set
- Writing/Reading data in HDF5 format
Deep Learning model training: Train a model using Keras with TensorFlow backend

Prerequisites

Docker CE
Nvidia-docker
GPU(at least 8GB)
ROS2. This project uses ROS2 Dashing.
Tensorflow, Keras
Python3
LGSVL Simulator

Clone the repository:

In the terminal at appropriate location run this command -

git clone https://github.com/dr563105/basicad_framework.git

Pull the latest Docker image

Then in the terminal run this command -

docker pull lgsvl/lanefollowing:latest

Run the simulator

Download the zipped modified simulator(filename: NewSimLFMay201.tar.gz) from my personal google drive.
Untar it and place the folder(NewSimLFMay201) along with the other folders from the repository.
Navigate to NewSimLFMay201->simulator->build.
Run simulator executable.
Click Open Browser
On the first login, create an account. Login. It should automatically download the default maps, vehicle, and simulation configurations.
Navigate to Vehiclestab and create a new setup by clicking Add new button.
Name the vehicle and use this as vehicle url. Click Submit.
Then click the wrench icon adjacent to the newly created vehicle profile. Make the Bridge Type as ROS2. Copy the contents from here to the Sensors field. Click Submit. Add/remove as necessary. Please note that the camera sensors transmit images in 1920x1080 resolution. So they are huge and can bottleneck port bandwidth.
Move to the Simulations tab, click Add new button and a popup menu will appear. In this menu's general tab, give a simulation name.
In the Maps & Vehicles tab, check interactive mode. Choose SanFrancisco as map.
Next in the Select Vehicles, select the vehicles tab configuration name from the dropdown and write localhost:9090 as IP address. Other tabs can be left empty for the moment. Click Submit.
Select the configured simulation and click play button. If there are no errors, the simulation should execute and in the simulation application we can see a vehicle spawned in SF map. Click the play button to make the LGSVL simulator ready.

Build ROS2 packages

Go to ROS2 workspace testoutLF->lanefollowing->ros2_ws.
Execute the build command in the terminal

docker-compose up build_ros

You should see two more folders -- build and log created.

Running the collect script

While in the ros2_ws, run this command in the terminal

docker-compose up collect

You should see the rosbridge is now connected. This can be verified by going to the settings in the LGSVL simulator app.
The script inside collect_script is copied into the ~/ros2_ws/lane_following/collect.py. The topic names of the nodes must match the topics mentioned in the vehicles tab of LGSVL WebUI.
To exit, simply use ctrl+c in the terminal.

Evaluation

To evaluate, the necessary evaluation script present inside evaluation_script must be copied into ~/ros2_ws/lane_following/drive.py.
The trained models and their architectures are available inside models_images_and_files folder in the repo.
Take the *.h5 hdf5 file and place it inside the folder ros2_ws/src/lane_following/model/. It contains the trained models accessed by the evaluation script.
The evaluation is run by executing this command in the terminal

docker-compose up drive_visual

A small window will pop up. Now, go back to LGSVL simulator application and gently accelerate using the Up arrow key. The vehicle will then autonomously navigate. Toggle traffic and change weather conditions as necessary.
To exit, simply use ctrl+c in the terminal.

Pre-processing and Training

For pre-processing, the scripts present inside preprocessing_script are used. For this step, a data folder is required. Because of data collected is of large size, this is not provided.
Similarly, for training, scripts inside training_script are used.
For convenience and CUDA dependencies, both the scripts are advised to be executed outside the docker.

Findings

Light plays an important factor in a vehicle recognising its surroundings.
RGB-Greyscale image is sufficient for steering.
Fusing techniques help in night-time driving and collision avoidance.

Things to improve

Data is imbalanced so appropriate cross validation techniques need to be applied.

Useful Links/Tips

Robotic Operating System(ROS)
LGSVL simulator
Docker CE and also its post installation.
steps
Nvidia docker
Use Conda environment for training. Conda automatically installs Tensorflow and CUDA dependencies and saves the headache of choosing which CUDA version is compatible with a particular version of tensorflow.
Always set ROS2 environment when running ROS related commands. source /opt/ros/dashing/setup.bash and from the ros2ws source install/setup.bash for local setup.
Thesis report folder contains the pdf of the report.
Further instructions can be found here and here.

deepakramani/basicad_framework