Instance segmentation for SfM point cloud.
Option 1 (recommended):
git clone https://github.com/ZhiangChen/semantic_SfM.git
cd semantic_SfM/semantic_SfM/ssfm/
python3 -m pip install -e .
Option 2:
python3 -m pip install semantic-SfM
Hardware requirements: This project utilizes the Segment Anything Model (SAM) for panoptic segmentation. GPUs are not required for SAM but are recommended to expedite inference speed. The other parts of this project use only CPUs. As a point cloud will be stored in memory, memory usage depends on the point cloud size.
files.py provides utility functions for file operation, such as point cloud and image files.
image_segmentation.py allows to select deep learning models for instance segmentation and panoptic segmentation.
probabilistic_projection.py projects point clouds to images and creates a probablistic semantics.
object_registration.py registers objects (instances and stuffs) in point clouds.
This repository supports SfM products from the following platforms:
Refer to the docs/projection_tutorial.md to obtain point clouds and camera parameters from WebODM or Agisoft. The point clouds should use the format of .las.
For image segmentation, we have used Segment Anything Model from Meta:
-
Projecting Point Clouds onto Image Planes: docs/projection_tutorial.md
-
Data Structure and Algorithms for Object Registration: docs/object_registration.md
-
Depth Image Rendering from Mesh and Depth Filtering: docs/depth_filtering.md
-
Camera Calibration Optimization: docs/camera_calibration_optimization.md
We provide jupyter notebooks (Semantic_SfM/projects/) as examples to use the semantic SfM workflow.
-
When I import point clouds into CloudCompare, the points look sparse with strips. How can I fix it? Such issues occur when point coordinates in CloudCompare are large. To fix the visualization issues, you can choose to automatically re-center the points when importing a point cloud.
-
My semantics in a resulting point cloud are shifted. Why does this happen? During bundle adjustment in SfM (e.g., photo alignment in Agisoft), cameras are automatically calibrated. However, the calibration is not always optimal. When the resulted
$c_x$ ,$c_y$ are large, the projected semantics on point clouds would be shifted. Please refer to this tutorial to optimize camera calibration.