Please acknowledge the authors and the following research paper:
"Unstructured point cloud semantic labeling using deep segmentation networks", A.Boulch and B. Le Saux and N. Audebert, Eurographics Workshop on 3D Object Retrieval 2017
Abstract and paper here.
This work is part of two projects:
- INACHUS is European FP7 project that aims to achieve a significant time reduction related to Urban Search and Rescue (USaR) phase by providing wide-area situation awareness solutions for improved detection and localisation of the trapped victims assisted by simulation tools for predicting structural failures and a holistic decision support mechanism incorporating operational procedures and resources of relevant actors.
- DeLTA is a research project hosted at ONERA, The French Aerospace Lab. Among its objectives are the development and the promotion of innovative machine learning based approaches for aerospace applications.
This code and weights are both released under a dual license, research and commercial. Basically the code is GPLv3 for open access (contact us for non open purposes) and the weights are released under Creative Commons BY-NC-SA (contact us for non open purposes). See the license.
The code is composed of two main parts: a C++ library and a python scripts. The C++ library permits to load the files at Semantic 3D format and compute the composite features as described in the paper.
The python scripts generates the views using the Pyqtgraph 3D viewer and predict the segmentation using TensorFlow.
- Cython
- PCL, for installation see pointclouds.org.
- OpenMP
Dependencies in the semantic3D_utils repository :
- NanoFlann: nanoflann.hpp
- Eigen
- TensorFlow: it is the deep learning framework used in this implementation (deprecated)
- Pytorch
- TQDM, Scipy, Numpy ...
To generate the C++/Python library.
cd semantic3D_utils
python setup.py install --home="."
It will build the library.
Note: in the setup.py, we have filled the path with the standard path for PCL installation libraries and headers on Ubuntu 16.04.
The Configuration file is a json file containing the parameters and paths:
{
"train_input_dir":"path_to_directory_TRAIN",
"test_input_dir":"path_to_directory_TEST",
"train_results_root_dir":"where_to_put_training_products",
"test_results_root_dir":"where_to_put_test_products",
"images_dir":"images",
"training": true,
"imsize":224,
"voxel_size":0.1,
"cam_number":10,
"create_mesh" : true,
"create_views" : true,
"create_images" : true,
"backend": "pytorch",
"vgg_weight_init":"path_to_vgg_weights",
"batch_size" : 24,
"learning_rate" : 1e-4,
"epoch_nbr" : 100,
"label_nbr" : 10,
"input_ch" : 3,
"train_rgb" : true,
"train_composite" : true,
"train_fusion" : true,
"saver_directory_rgb" : "path_to_rgb_model_directory",
"saver_directory_composite" : "path_to_composite_model_directory",
"saver_directory_fusion" : "path_to_fusion_model_directory",
"output_directory":"path_to_output_product_directory"
}
It is used in the python scripts, to avoid code recopy.
For the training and testing dataset, the point cloud decimation, views and images generation are called with:
python3 sem3d_gen_images.py --config config.json
To train the models (rgb, composite and fusion) from scratch, run:
python3 sem3d_train.py --config config.json
The semantic predictions on images and back-projection on the decimated clouds can be called using:
python3 sem3d_test_backproj.py --config config.json
Finally to generate the files at the Semantic 3D format and assign a label to each point of the original point cloud, run;
python3 sem3d_test_to_sem3D_labels.py --config config.json
The new code has not been tested with tensorflow. Updates on the tensorflow code are welcome. It is also possible to revert to a previous commit to get a working version.
For the old Tensorflow version, the pre-trained weights are available:
Fusion model (to be used with RGB and Composite)
For training from scratch, if needed, the pretrained weights of VGG16, from the caffe model are available here