Yuchen Zhou * 1, Jiayuan Gu * 1, Tung Yen Chiang1, Fanbo Xiang1, Hao Su1,2
UC San Diego, Hillbot
* Equal Contribution
Our model support mesh segmentation by sampling points from the mesh and propagate segmentation labels back to the mesh. We deployed a huggingface demo at 🤗demo🤗. We also provide local mesh segmentation demo at https://github.com/zyc00/point-sam-demo. Some meshes in glb format are provided in examples.
The code requires python>=3.8, timm>=0.9.0, pytorch>=2.1.0, torchvision>=0.16.0. Please follow the offcial guide to install Pytorch, timm and TorchVision dependencies. We also recommend compiling third partiy modules with g++=9.3.0.
Install third party modules.
# Install g++=9.3.0 in conda environment by
conda install gxx_linux-64=9.3.0
# Install torkit3d
git submodule update --init third_party/torkit3d &&
FORCE_CUDA=1 pip install third_party/torkit3d &&
# Install apex
git submodule update --init third_party/apex &&
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" third_party/apex
Install Point-SAM from GitHub
pip install git+https://github.com/zyc00/Point-SAM.git
or clone the repository and install with
git clone https://github.com/zyc00/Point-SAM.git &&
cd Point-SAM && pip install . && cd ..
We provide pretrained checkpoint for Point-SAM with ViT-L on HuggingFace. After downloading the checkpoint, you can inference with the following codes. For different numbers of points, we allow custom group_number and group_size. The default setting is group_number=512 and group_size=64, and we suggest setting group_number=2048 and group_size=256 while the number of points > 100k.
from point_sam import build_point_sam
model = build_point_sam(ckpt_path, group_number, group_size)
model.set_pointcloud(coords, colors)
model.predict_masks(prompt_coords, prompt_labels)
We only support meshes in GLB format. If your mesh is not in GLB format, please convert it first. For our demo, you can simply drag a mesh into the browser, and the backend will encode the point cloud. Once encoding is complete, the camera view will adjust for optimal viewing. Then, you can interact with the demo by clicking on the mesh surface.
The performance of the Hugging Face demo is limited by data transfer and a shared GPU backend. To experience better performance, try running our local demo on your device by executing python app.py.
We provide a local demo for Point-SAM in demo. Some point cloud examples are provided in demo/static/models. If you want to try our demo, first install backend dependencies.
pip install flask flask-cors
Run the backend with following commands, you can change --host and --port to your IP address.
python demo/app.py --host localhost --port 5000 --pointcloud scene.ply --checkpoint ./pretrained/model.safetensors
# python demo/app.py --host localhost --port 5000 --pointcloud rhino.ply --checkpoint ./pretrained/model.safetensors
# python demo/app.py --host localhost --port 5000 --pointcloud tiko_10000_points.ply --checkpoint ./pretrained/model.safetensors
# python demo/app.py --host localhost --port 5000 --pointcloud tiko_50000_points.ply --checkpoint ./pretrained/model.safetensors
We refer to SAM, Uni3D and OpenShape while developing Point-SAM. Thanks for these awesome works. We also thank @YouJiacheng for providing constructive suggestions for the data engine design.
@misc{zhou2024pointsampromptable3dsegmentation,
title={Point-SAM: Promptable 3D Segmentation Model for Point Clouds},
author={Yuchen Zhou and Jiayuan Gu and Tung Yen Chiang and Fanbo Xiang and Hao Su},
year={2024},
eprint={2406.17741},
archivePrefix={arXiv},
primaryClass={cs.CV}
url={https://arxiv.org/abs/2406.17741},
}