LiteFlowNet2

This repository (https://github.com/twhui/LiteFlowNet2) is the offical release of LiteFlowNet2 for my paper A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization in TPAMI 2020 ( https://ieeexplore.ieee.org/document/9018073). The arXiv pre-print is available here.

LiteFlowNet2, another lightweight convolutional network, is evolved from our earlier work in CVPR 2018 (https://github.com/twhui/LiteFlowNet) to better address the problem of optical flow estimation by improving flow accuracy and computation time. Comparing to our earlier work, LiteFlowNet2 improves the optical flow accuracy on Sintel clean pass by 23.3%, Sintel final pass by 12.8%, KITTI 2012 by 19.6%, and KITTI 2015 by 18.8%. Its runtime is 2.2 times faster!

	Sintel Clean Testing Set	Sintel Final Testing Set	KITTI12 Testing Set (Out-Noc)	KITTI15 Testing Set (Fl-all)	Model Size (M)	Runtime* (ms) GTX 1080
FlowNet2 (CVPR17)	4.16	5.74	4.82%	10.41%	162	121
PWC-Net+	3.45	4.60	3.36%	7.72%	8.75	40
LiteFlowNet2 (TPAMI)	3.48	4.69	2.63%	7.62%	6.42	40

Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.

License and Citation

This software and associated documentation files (the "Software"), and the research paper (A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization) including but not limited to the figures, and tables (the "Paper") are provided for research purposes only and without any warranty. Any commercial use requires my consent. When using any parts of the Software or the Paper in your work, please cite the following paper:

@InProceedings{hui20liteflownet2,    
 author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
 title = {A {L}ightweight {O}ptical {F}low {CNN} - {R}evisiting {D}ata {F}idelity and {R}egularization}, 
 journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
 year = {2020},    
 url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
}

@InProceedings{hui18liteflownet,    
 author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
 title = {LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation},    
 booktitle  = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},    
 year = {2018},    
 pages = {8981--8989},
 url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
}

Prerequisite and Compiling

LiteFlowNet2 uses the same Caffe package as LiteFlowNet. Please refer to the details in LiteFlowNet GitHub repository.

Training

Please refer to the training steps in LiteFlowNet GitHub repository and adopt the training prtocols in LiteFlowNet2 paper.

Trained models

The trained models (LiteFlowNet2-ft-sintel, LiteFlowNet2-ft-kitti) are available in the folder /models/trained. Untar the files to the same folder before you use it.

LiteFlowNet2-ft-sintel: Model used for Sintel benchmark.

LiteFlowNet2-ft-kitti: Model used for KITTI benchmark.

Testing

Open the testing folder

$ cd LiteFlowNet2/models/testing

Create a soft link in the folder /testing

$ ln -s ../../build/tools bin

Replace MODE in ./test_MODE.py to batch if all the images has the same resolution (e.g. Sintel dataset), otherwise replace it to iter (e.g. KITTI dataset).
Replace MODEL in line 10 (cnn_model = 'MODEL') of test_MODE.py to one of the trained models (e.g. LiteFlowNet2-ft-sintel).
Run the testing script. Flow fields (MODEL-0000000.flo, MODEL-0000001.flo, ... etc) are stored in the folder /testing/results having the same order as the image pair sequence.

$ test_MODE.py img1_pathList.txt img2_pathList.txt results

Reimplementation in TensorFlow

A TensorFlow-based reimplementation of LiteFlowNet2 is also available at https://github.com/rogerhcheng/LiteFlowNet2-TF2.

491913145/LiteFlowNet2