/opticalflow-autoflow

Primary LanguageJupyter NotebookApache License 2.0Apache-2.0

Optical Flow by Google Research VisCam

Papers

Dataset, code, and checkpoints for two optical flow papers from Google Research Viscam.

Disentangling Architecture and Training for Optical Flow
Deqing Sun*,T, Charles Herrmann*, Fitsum Reda, Michael Rubinstein, David Fleet, William T. Freeman
Google Research
In ECCV 2022. * denotes equal technical contribution, T denotes project lead.

Teaser image.

Left: Large improvements with newly trained PWC-Net, IRR-PWC and RAFT (left: originally published results in blue; results of our newly trained models in red). The newly trained RAFT is more accurate than all published methods on KITTI 2015 at the time of writing.

Right: Visual comparison on a Davis sequence between the original [43] and our newly trained PWC-Net and RAFT, shows improved flow details, e.g. the hole between the cart and the person at the back. The newly trained PWC-Net recovers the hole between the cart and the front person better than RAFT.

AutoFlow: Learning a Better Training Set for Optical Flow
Deqing Sun, Daniel Vlasic , Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T. Freeman, Ce Liu
Google Research
In CVPR 2021 (Oral).

Inference

Pre-trained Models

Create a directory where you can keep large files. Ideally, not in this directory.

> mkdir <pretrained_models>

Download pre-trained TF2 Saved Models from google drive and put into <pretrained_models>.

The downloaded folder should have the following structure:

pretrained_models/
├── eccv_disentangling_architecture_and_train/
│   ├── pwc_it_pre/
│   ├── pwc_it_ft/
│   ├── raft_it_pre/
│   ├── raft_it_ft/

Running the optical flow network

To run inference on examples frames we will use the "OpticalFlow-Inference.ipynb" colab in colabs/. To run the colab, make sure that you have installed all the required packages with the following command.

> pip install -r requirements.txt

Then make sure you have the pre-trained models downloaded to the correct folder as shown in the section above. Then start a colab kernel using the follow command.

> jupyter notebook --NotebookApp.allow_origin='https://colab.research.google.com' --port=8888   -NotebookApp.port_retries=0

Then navigate to the "OpticalFlow-Inference.ipynb" and run all the cells.

Inference example.

Example of retrained PWC-Net from our most recent submission (run in this inference colab)

Training

The src/ directory currently contains the augmentation module (the Improved Training result uses the "pwc" augmentation from the augmentations module). The full train loop will be posted soon.

Citation

If you find this useful in your works, please acknowledge it appropriately by citing:

@article{sun2022disentangling,
  title={Disentangling Architecture and Training for Optical Flow},
  author={Sun, Deqing and Herrmann, Charles and Reda, Fitsum and Rubinstein, Michael and Fleet, David and Freeman, William T},
  journal={arXiv preprint arXiv:2203.10712},
  year={2022}
}

@inproceedings{sun2021autoflow,
  title={Autoflow: Learning a better training set for optical flow},
  author={Sun, Deqing and Vlasic, Daniel and Herrmann, Charles and Jampani, Varun and Krainin, Michael and Chang, Huiwen and Zabih, Ramin and Freeman, William T and Liu, Ce},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10093--10102},
  year={2021}
}

Contact: Deqing Sun (deqingsun @ google.com) and Charles Herrmann (irwinherrmann @ google.com)

Coding style

  • 2 spaces for indentation
  • 80 character line length
  • PEP8 formatting

Disclaimer

This is not an officially supported Google product.