This research, based on the master thesis of the author, is focused on the development of a NN-based approach for Object Detection and 6 DoF Pose Estimation with ToF Cameras for space applications. The aim is that of designing a NN-based approach able to detect a part of a satellite (such as a gripping interface) and estimate its 6 DoF pose to support on-orbit servicing missions.
Please note that the project is based on Python v3.8.12, TensorFlow v2.6.0 and PyTorch v1.9.1, and it is built upon MacOS BigSur v11.3.1.
main.py is the starting script: it retrieves the camera interface available for recordings and it begins the data acquisition.
In order to make this code works, after having downloaded the software from the official site, the right .zip folder has to be extracted and placed into this one, so that _roypy.so image can be found by the scripts.
utils folder contains the scripts for acquiring data in real-time, using both Open3D for point cloud drawings and OpenCV for depth maps and amplitude images visualization.
camera_imagedata.py script captures image data, filling OpenCV images and displaying data with HighGUI.
OpenCV can be downloaded here.
camera_3d.py script captures depth maps and visualizes 3D Point Clouds dinamically changing.
Please find more details about the library here.
camera_imagedata_3d.py script captures both depth and amplitude data, starting different threads to simultaneously show images through OpenCV library and 3D point clouds through Open3D.
mpi_denoising folder contains the code for denoising the data acquired through the camera. In particular, shot and MPI denoising are performed.
Two state-of-the-art networks are implemented, namely Coarse-Fine CNN and SHARPNet.
coarse_fine_cnn folder contains a model that is a hybrid implementation of the network (and the data augmentation strategy) described in this paper and the one available at this paper, and it is built thanks to the useful insights provided by the author of the papers, Gianluca Agresti.
Please read the paper(s) for more information.
sharp_net folder contains the implementation of the model described in this paper. Since we are interested in a single modulation frequency method, this is the approach that is finally used in this work.
Please find more details about the algorithm in the linked paper.
pcd_registration folder contains the code for the point-cloud registration between a 3D satellite model target and the source acquired one. Three state-of-the-art models are currently taken into account to perform this final task, namely Fast Global Registration (FGR), FPFH + RANSAC Registration (FRR), and Feature-Metric Registration (FMR).
fast_global_registration.py implements the fast global registration approach, that is described here. It achieves state-of-the-art results for what concerns Optimization-Based methods.
global_registration.py approach uses FTFH features extraction as described here, followed by the classical RANSAC method presented in this paper. This model has been chosen to be the representative of Feature-Learning methods, due to its simplicity and yet its effectiveness.
fmr folder contains the code for the implementation of the algorithm described here. The strength of this method relies on its robustness to noise, outliers and density difference (in contrast with "geometric" methods as the ones previously introduced), its fastness and it high accuracy.
Please find more information about the model in the relative paper.
To reduce the complexity of the proposed SHARP-Net, two variants are proposed, ToF-KPN (based on a U-Net model built upon an autoencoder with skip connections) and a reduced version of SHARP-Net without the last two blocks.
The following images show the results of the denoising approaches: ToF-KPN always obtains the worst outcomes, while the two variants of SHARP-Net get very similar ones.
By considering the execution times, ToF-KPN is always the fastest, being the simplest model. The original SHARP-Net is the slowest, while its smaller variant achieves very good results in almost half the time required by SHARP-Net.
Thus, its low complexity, its online performance and the optimal results make it a perfect candidate for this work.
Several starting poses for the source and different sizes for the target are considered to evaluate the performance and the robustness of the registration approaches. Feature-Metric Registration clearly outperforms all the others regardless of the starting position of the source point cloud and the dimensionality of the target one.
FMR always gets the best results while still being incredibly fast: it only takes less than 0.4 seconds to converge with relatively small point clouds (as the ones obtained by the camera used in this work).
Its optimal results, coupled with the low complexity and the online performance, definitely prove the choice that has been made about this algorithm.