In this project, I used Kitti360 dataset to give pointcloud semantic labels using segmentation obtained from a camera image of the scene. I used an implementation of segformer to generate semantic labels on the image. After applying semantic labels on multiple pointclouds, I used ICP registration to generate the mapping of the complete scene.
This repository does the following tasks.
- Intrinsic calibration of perspective camera (Point Gray Flea 2).
- Extrinsic calibration of the perspective camera with a Velodyne Lidar (HDL-64E).
- Projecting lidar points on to the image plane (run
python src/utils.py). - Generating a colored pointcloud (run
python src/gen_color_pcd.py). - Applying semantic labels on the pointcloud and registering multiple pointclouds using ICP based registration(run
python src/pointcloud_segmentation.py).
Refer to this page to install openMMlab's segmentation repository. Download and paste the desired model in the config folder. Change the segment_image.py file accordingly. Install OpenCV on python.
- To run the complete pipeline with segmentation and ICP registration:
python src/pointcloud_segmentation.py
- To generate colored pointcloud:
python src/gen_color_pcd.py
- To project lidar points onto the image:
python src/utils.py
Figure 1: Original Pointcloud
Figure 2: Poincloud Projection on Image Plane
Figure 3: Colored Pointcloud
Figure 4: Segmented Pointcloud
Figure 5: ICP Registration demo