/flownet2-tf

FlowNet 2.0 (tf): modified for optical flow interpolation (only FlowNetS ATM)

Primary LanguagePythonMIT LicenseMIT

TODO: just presented the MsC thesis, link to the thesis and helpful scripts + proper README.md incoming the following weeks!

FlowNet2 (TensorFlow) adapted for Optical Flow Interpolation

This fork stems from the work done by @sampepose in flownet2-tf, and adapts the FlowNetS architecture for the problem of Optical Flow Interpolation. In that regard, several changes have been performed in order to add validation via TF-Slim, new training policies (CLR and One Cycle), new optimisers (AdamW) and other utilities (functions to compute optical flow error metrics, etc.). More details will be provided in the near-future. This work has been done in the framework of a MsC in Computer Vision thesis. We will link to the report as soon as it is available to the public.

TODO: shortly we will update the scripts used for training, testing and learning rate range tests

These are vert handy to change the parameters in better-structured shell scripts instead of having to type them in the command line for each experiment.

What follows is the original repository README.md with instructions on installation and basic usage (this will be deleted once we have a detailed README.md that considers the changes briefly mentioned above).

FlowNet2 (TensorFlow)

This repo contains FlowNet2[1] for TensorFlow. It includes FlowNetC, S, CS, CSS, CSS-ft-sd, SD, and 2.

Installation

pip install enum
pip install pypng
pip install matplotlib
pip install image
pip install scipy
pip install numpy
pip install tensorflow

Linux: sudo apt-get install python-tk

You must have CUDA installed: make all

Download weights

To download the weights for all models (4.4GB), run the download.sh script in the checkpoints directory. All test scripts rely on these checkpoints to work properly.

Flow Generation (1 image pair)

python -m src.flownet2.test --input_a data/samples/0img0.ppm --input_b data/samples/0img1.ppm --out ./

Available models:

  • flownet2
  • flownet_s
  • flownet_c
  • flownet_cs
  • flownet_css (can edit test.py to use css-ft-sd weights)
  • flownet_sd

If installation is successful, you should predict the following flow from samples/0img0.ppm: FlowNet2 Sample Prediction

Training

If you would like to train any of the networks from scratch (replace flownet2 with the appropriate model):

python -m src.flownet2.train

For stacked networks, previous network weights will be loaded and fixed. For example, if training CS, the C weights are loaded and fixed and the S weights are randomly initialized.

Fine-tuning

TODO

Benchmarks

Benchmarks are for a forward pass with each model of two 512x384 images. All benchmarks were tested with a K80 GPU and Intel Xeon CPU E5-2682 v4 @ 2.30GHz. Code was executed with TensorFlow-1.2.1 and python 2.7.12 on Ubuntu 16.04. Resulting times were averaged over 10 runs. The first run is always slower as it sets up the Tensorflow Session.

S C CS CSS SD 2
First Run 681.039ms 898.792ms 998.584ms 1063.357ms 933.806ms 1882.003ms
Subsequent Runs 38.067ms 78.789ms 123.300ms 161.186ms 62.061ms 276.641ms

Sources

[1] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, T. Brox FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks, IEEE Conference in Computer Vision and Pattern Recognition (CVPR), 2017.