Data Fusion and Multi-Task Learning

This project intends to study the Data Fusion and Multi-Task Learning applied to the detection of dangerous situations during drive.

The project is divided in three parts:

Dataset Generation:
- Uses CARLA simulator to generate the dataset
Opponent Detection
- Convolutional Neural Networks
Situation Evaluation
- Based on the opponent detection, car position

Requirements

Python3 and libraries
Tensorflow 2.0 and PyTorch
Carla Simulator 9.6.0+

Install python libraries:

python3 -m pip install pyqtgraph pygame matplotlib h5py opencv-python future numpy jupyter PyQt5 open3d

1. Dataset Generation

Carla Simulator Setup

Carla Simulator is used to generate the dataset to train and test the network models.

Download and install:

mkdir -p simulator/carla
cd simulator/carla
wget http://carla-assets-internal.s3.amazonaws.com/Releases/Linux/CARLA_0.9.6.tar.gz
tar -xf CARLA_0.9.6.tar.gz

Python scripts for data generation

Generating Data

Start Carla:

cd scripts
./run_carla_server.sh

One LiDAR sensor and 1-4 cameras are set up in the EGO vehicle.

Run Scripts:

cd scripts
python3 dataset_generation.py

The data generated will be stored on the "./dataset" folder.

Visualizing the generated data:

cd scripts
python3 dataset_visualization.py