This repo contains experimental code for the CARLA Simulation Environment. The supported features are below:
- Collect sensor data from the CARLA simulation environment (data_collector.py)
- Auto label bounding boxes and labels for objects in the CARLA simulation environment (auto_labeler.py)
- Train a PyTorch neural network to detect the bounding box locations of vehicles, pedestrains, and traffic signs in the CARLA simulation environment (object_detection_nn.py)
- Run real-time bounding box object detection, depth estimation, and lane line/drivable space detection in the CARLA simulation environment (perception_algorithm.py will become perception_stack.py)
- Install Carla: https://carla.readthedocs.io/en/latest/start_quickstart/
- Install Python: https://www.python.org/downloads/
- Install Python Dependencies: Navigate to root of this repo and run command
pip install -r requirements.txt
This quickstart guide will walk through working with all the major features of this repo (assumes requirements are met):
- Starting the CARLA Simulation Environment Server:
- In a command prompt, navigate to the directory containing CARLA.exe and run the below command:
CarlaUE4.exe -windowed -carla-server -fps=20
- In a command prompt, navigate to the directory containing CARLA.exe and run the below command:
- Collecting Sensor Data From CARLA
- Navigate to the root of this repo, and run data_collector.py to collect sensor data from CARLA (modify the global variables to control number of episodes and frames to capture) using the following command:
python data_collector.py --autopilot --images-to-disk
- Navigate to the root of this repo, and run data_collector.py to collect sensor data from CARLA (modify the global variables to control number of episodes and frames to capture) using the following command:
- Auto-label Object Bounding Boxes:
- Run auto_labeler.py to generate JSON files with object labels and bounding box coordinates from the captured data_collection images (update object class labels to detect if desired) using the following command:
python auto_labeler.py
- Run auto_labeler.py to generate JSON files with object labels and bounding box coordinates from the captured data_collection images (update object class labels to detect if desired) using the following command:
- Train Bounding Box Objection Detection Neural Network Model:
- Run object_detection_nn.py on the captured and labeled images to train a neural network to detect object bounding box locations (modify the global variables as desired) using the following command:
python object_detection_nn.py
- Run object_detection_nn.py on the captured and labeled images to train a neural network to detect object bounding box locations (modify the global variables as desired) using the following command:
- Real-time Perception Stack:
- Run perception_algorithm.py to run real-time object detection, distance estimation, and lane line detection using the following command:
python perception_algorithm.py
- Run perception_algorithm.py to run real-time object detection, distance estimation, and lane line detection using the following command:
This section describes the features of this repo as well as future features.
- CARLA sensor data collection using the CARLA API - [COMPLETE! data_collector.py]
- Auto labeling CARLA object's bounding boxes and class using the semantic segmentation image - [COMPLETE! auto_labeler.py]
- Customizable Transfer Learning from VGG16 COCO pre-trained model using PyTorch
The following perception stack items can be executed in real-time detection
- Object bounding box location from trained model
- Distance estimation of detected objects using the disparity image
- Lane detection using semantic segmentation image and edge detection
- Vehicle speed and trajectory estimation using Optical Flow
- Vehicle localization using Visual Odometry
- 3D, 360 Degree Environment Map using Image Stitching
- Improvements to customizability of perception stack (to be defined)
Future work is planned to create a control model for use in this Autonomous CARLA stack.
- Kinematic, Dynamic, Longitudnal, and Lateral Control Models
Future work is planned to create a state estimation and localization algorithm for use in this Autonomous CARLA stack.
- State estimation using variations of the Kalman Filter and vehicle sensors
Future work is planned to create a mapping and path planning algorithm for use in this Autonomous CARLA stack.
- Vehicle mapping and path planning using graphs and state machines
Below are links to code and theory references used in this project:
- Visual Perception for Self-Driving Cars Coursera Course by University of Toronto: https://www.coursera.org/learn/visual-perception-self-driving-cars
- CARLA Documentation, Examples, and API: https://carla.org/