SeparableFlow

Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation

Building Requirements:

gcc: >=5.3
GPU mem: >=5G (for testing);  >=11G (for training)
pytorch: >=1.6
cuda: >=9.2 (9.0 doesn’t support well for the new pytorch version and may have “pybind11 errors”.)
tested platform/settings:
  1) ubuntu 18.04 + cuda 11.0 + python 3.6, 3.7
  2) centos + cuda 11 + python 3.7

Environment:

conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch-lts -c nvidia
conda install matplotlib tensorboard scipy opencv
pip install einops opencv-python pypng

How to Use?

Step 1: compile the libs by "sh compile.sh"

Change the environmental variable ($PATH, $LD_LIBRARY_PATH etc.), if it's not set correctly in your system environment (e.g. .bashrc). Examples are included in "compile.sh".

Step 2: download and prepare the training dataset or your own test set.

Step 3: revise parameter settings and run "train.sh" and "evaluate.sh" for training, finetuning and prediction/testing. Note that the “crop_width” and “crop_height” must be multiple of 64 during training.

Demo example: (use "sintel" or "universal" for other unseen datasets):
$ python demo.py --model checkpoints/sepflow_universal.pth --path ./your-own-image-folder

Pretrained models:

things	sintel	kitti	universal
Google Drive	Google Drive	Google Drive	Google Drive
Baidu Yun (password: 9qcd)	Baidu Yun (password: m1xs)	Baidu Yun (password: sg46)	Baidu Yun (password: 2has)

These pre-trained models perform a little better than those reported in our original paper. "universal" is trained on a mixture of synthetic and real datasets for cross-domain generalization.

Leadboards	Sintel clean	Sintel final	KITTI
RAFT baseline	1.94	3.18	5.10
Orginal paper	1.50	2.67	4.64
This new implementation	1.49	2.64	4.53

Standard two-frame evaluations without previous video frames for "warm start".

Reference:

If you find the code useful, please cite our paper:

@inproceedings{Zhang2021SepFlow,
  title={Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation},
  author={Zhang, Feihu and Woodford, Oliver J. and Prisacariu, Victor Adrian and Torr, Philip H.S.},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021}
  pages={10807-10817}
}

The code is implemented based on https://github.com/feihuzhang/DSMNet and https://github.com/princeton-vl/RAFT. Please also consider citing:

@inproceedings{zhang2019domaininvariant,
  title={Domain-invariant Stereo Matching Networks},
  author={Feihu Zhang and Xiaojuan Qi and Ruigang Yang and Victor Prisacariu and Benjamin Wah and Philip Torr},
  booktitle={Europe Conference on Computer Vision (ECCV)},
  year={2020}
}
@inproceedings{teed2020raft,
  title={RAFT: Recurrent All Pairs Field Transforms for Optical Flow},
  author={Zachary Teed and Jia Deng},
  booktitle={Europe Conference on Computer Vision (ECCV)},
  year={2020}
}