HPLFlowNet

This is the code for HPLFlowNet, a hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. The code is developed and maintained by Xiuye Gu.

Prerequisites

Our model is trained and tested under:

Python 3.5.2 (testing under Python 3.6.5 also works)
NVIDIA GPU + CUDA CuDNN
PyTorch (torch == 0.4.0) (Should also support PyTorch 1.0 now)
Numba (numba == 0.38.1)
You may need to install cffi.
Mayavi for visualization.
Installation on Ubuntu:

pip3 install https://download.pytorch.org/whl/cu90/torch-0.4.0-cp35-cp35m-linux_x86_64.whl
pip3 install numba
pip3 install cffi
sudo apt-get -y install python-vtk
sudo pip3 install mayavi
sudo apt-get install python3-pyqt5
sudo pip3 install PyQt5

Data preprocess

FlyingThings3D: Download and unzip the "Disparity", "Disparity Occlusions", "Disparity change", "Optical flow", "Flow Occlusions" for DispNet/FlowNet2.0 dataset subsets from the FlyingThings3D website (we used the paths from this file, now they added torrent downloads) . They will be upzipped into the same directory, RAW_DATA_PATH. Then run the following script for 3D reconstruction:

python3 data_preprocess/process_flyingthings3d_subset.py --raw_data_path RAW_DATA_PATH --save_path SAVE_PATH/FlyingThings3D_subset_processed_35m --only_save_near_pts

KITTI Scene Flow 2015 Download and unzip KITTI Scene Flow Evaluation 2015 to directory RAW_DATA_PATH. Run the following script for 3D reconstruction:

python3 data_preprocess/process_kitti.py RAW_DATA_PATH SAVE_PATH/KITTI_processed_occ_final

Get started

Setup:

cd models; python3 build_khash_cffi.py; cd ..

Trained models

Out trained model can be downloaded in the trained_models folder.

Inference

Set data_root in the configuration file to SAVE_PATH in the data preprocess section. Set resume to be the path of your trained model or our trained model in trained_models. Then run

python3 main.py configs/test_xxx.yaml

Current implementation only supports batch_size=1.

Train

Set data_root in the configuration file to SAVE_PATH in the data preprocess section. Then run

python3 main.py configs/train_xxx.yaml

Visualization

If you set TOTAL_NUM_SAMPLES in evaluation_bnn.py to be larger than 0. Sampled results will be saved in a subdir of your checkpoint directory, VISU_DIR.

Run

python3 visualization.py VISU_DIR

Citation

If you use this code for your research, please cite our paper.

@inproceedings{HPLFlowNet,
  title={HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for
Scene Flow Estimation on Large-scale Point Clouds},
  author={Gu, Xiuye and Wang, Yijie and Wu, Chongruo and Lee, Yong Jae and Wang, Panqu},
  booktitle={Computer Vision and Pattern Recognition (CVPR), 2019 IEEE International Conference on},
  year={2019}
}

Acknowledgments

Our permutohedral lattice implementation is based on Fast High-Dimensional Filtering Using the Permutohedral Lattice. The BilateralNN implementation is also closely related. Our hash table implementation is from khash-based hashmap in Numba.

laoreja/HPLFlowNet