/CrowdFlow

Optical Flow Dataset and Benchmark for Visual Crowd Analysis

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

Dataset samples

TUB CrowdFlow Dataset

Optical Flow Dataset and Evaluation Kit for Visual Crowd Analysis developed at Communication Systems Group at TU-Berlin desciribed in the AVSS 2018 paper Optical Flow Dataset and Benchmark for Visual Crowd Analysis or TUBCrowdFlow@arxiv.org.

The Dataset contains 10 sequences showing 5 scenes. Each scene is rendered twice: with a static point of view and a dynamic camera to simulate drone/UAV based surveillance. We render at HD resolution (1280x720) at 25 fps, which is typical for current commercial CCTV surveillance systems. The total number of frames 3200.

For each sequence we provide the following ground-truth data:

  • Optical flow fields
  • Person trajectories (up to 1451)
  • Dense pixel trajectories

This evaluation framework is released under the MIT License (details in LICENSE). If you use the dataset or evaluation kit or think our work is useful in your research, please consider citing:

@INPROCEEDINGS{TUBCrowdFlow2018,
	AUTHOR = {Gregory Schr{\"o}der and Tobias Senst and Erik Bochinski and Thomas Sikora},
	TITLE = {Optical Flow Dataset and Benchmark for Visual Crowd Analysis},
	BOOKTITLE = {IEEE International Conference on Advanced Video and Signals-based Surveillance},
	YEAR = {2018},
}

Download the dataset via the following direct link

The password is the case sensitive name of the repository.

Unpack the dataset by:

sudo apt-get install unrar
unrar x TUBCrowdFlow

The TUB CrowdFlow dataset is made available for academic use only. If you wish to use this dataset commercially please contact sikora@nue.tu-berlin.de.

Contact

If you have any questions or encounter problems regarding the method/code or want to send us your optical flow benchmark results feel free to contact me at tobias.senst@gmail.com

Installation

Minimum required python version: 3.5

Install dependencies on Ubuntu:

sudo apt-get install python3-dev python3-virtualenv virtualenv

Create a virtual environment and install python requirements:

virtualenv -p python3 crowdflow_env
source crowdflow_env/bin/activate
pip3 install numpy progressbar2 opencv-contrib-python

Evaluation Framework

To evaluate an optical flow method with the providen framework perform these step:

  • create a new directory in the /TUBCrowdFlow/estimate directory.
  • compute flow fields and save them in .flo fileformat with the structure given in by the /TUBCrowdFlow/images directory. For example optical flow results from the image pair /TUBCrowdFlow/images/IM01/frame_0000.png and /TUBCrowdFlow/images/IM01/frame_0001.png must be stored as `/estimate/[mymethod]/images/IM01/frame_0000.flo
  • run opticalflow_evaluate.py to compute EPE and R2 short-term metrics.
  • run trajectory_evaluate.py to compute tracking accuracy long-term metrics.

Optical Flow Samples

opticalflow_estimate.py <dataset_root_path> <flow_method_name_1> <flow_method_name_2> ...

With the following program optical flow fields for the TUB CrowdFlow dataset will be estimated with Dual-TVL1.

source crowdflow_env/bin/activate
python3 opticalflow_estimate.py TUBCrowdFlow/ dual farneback plk

The optical flow files will be stored in the directory /estimate/dual/ .

Short-Term Evaluation

Short-term evaluation performs classical approch for optical flow evaluation, i.e. measures based on ground-truth optical flow fields (e.g. end-point error, RX measures)

opticalflow_evaluate.py <dataset_root_path> <dir_name_method_1> <dir_name_method_2> ... <dir_name_method_n>

Example:

source crowdflow_env/bin/activate
python3 opticalflow_evaluate.py TUBCrowdFlow/ dual plk farneback

After execution the file short_term_results.tex will contain the evaluation results (method 1 - method n) in form of a latex table. short_term_results.pb will contain the evaluation results stored with pickle.

Long-Term Evaluation

Long-term evaluation performs evaluation based on ground-truth trajectories, i.e. person trajectories and dense pixel trajectories (see paper).

trajectory_evaluate.py <dataset_root_path> <dir_name_method_1> <dir_name_method_2> ... <dir_name_method_n>

Example:

source crowdflow_env/bin/activate
python3 trajectory_evaluate.py TUBCrowdFlow/ dual plk farneback 

After execution the file long_term_results.tex will contain the evaluation results (method 1 - method n) in form of a latex table. long_term_results.pb will contain the evaluation results stored with pickle.

Results

To assess the quality of the optical flow we propose to use two types of metrics: i) common optical flow metrics, i.e. average endpoint error (EPE) and percentage of erroneous pixel (RX) and ii) long-term motion metrics based on trajectories. An detailed overview of the optical flow parameters can be found in the document: Supplemental_materials.pdf).

Common optical flow metrics (short-term)

FG (Static) FG (Static) BG (Static) BG (Static) FG (Dynamic) FG (Dynamic) BG (Dynamic) BG (Dynamic) FG(Avg.) FG(Avg.) BG(Avg.) BG(Avg.) Avg. Avg.
EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] t[sec]
FlowFields (Bailer2015) 0.756 8.27 0.213 2.79 1.069 14.92 2.571 51.42 0.913 11.595 1.392 27.10 0.915 11.74 43.53
RIC (Hu2017) 0.859 8.64 0.243 3.31 1.166 15.69 2.623 53.58 1.013 12.164 1.433 28.45 1.015 12.32 8.30
CPM (Li2018) 0.701 7.09 0.247 3.63 1.026 13.94 2.585 51.78 0.864 10.517 1.416 27.71 0.868 10.69 14.74
DeepFlow (Weinzaepfel2013) 0.629 6.19 0.237 3.67 1.005 13.95 2.594 51.67 0.817 10.069 1.416 27.67 0.822 10.25 39.63
RLOF6 (Geistert2016) 0.753 8.61 0.315 5.00 1.088 15.61 2.655 53.47 0.921 12.112 1.485 29.23 0.924 12.27 1.49
RLOF10 (Geistert2016) 0.772 8.80 0.324 5.10 1.104 15.80 2.658 53.60 0.938 12.303 1.491 29.35 0.941 12.46 0.80
DIS4 (Kroeger2016) 0.627 5.72 0.356 5.85 0.928 11.86 2.665 53.67 0.777 8.790 1.511 29.76 0.784 9.01 1.70
DIS2 (Kroeger2016) 1.441 20.40 0.528 8.24 1.726 27.41 3.001 64.01 1.583 23.903 1.765 36.13 1.579 23.92 0.28
Farneback (Farneback2003) 0.737 7.21 0.441 7.30 0.996 12.67 2.491 50.60 0.867 9.940 1.466 28.95 0.872 10.13
Sparse to Dense PLK (Bouguet2000) 0.793 8.07 0.563 9.12 1.041 13.24 2.875 56.29 0.917 10.653 1.719 32.71 0.925 10.88

Tracking Accuracy (long-term)

Dense Trajectories

IM01 (Dyn) IM02 (Dyn) IM03 (Dyn) IM04 (Dyn) IM05 (Dyn) Avg.
FlowFields (Bailer2015) 70.63 61.79 56.69 45.93 71.46 68.35 42.27 37.63 65.15 59.61 57.95
RIC (Hu2017) 74.39 69.41 58.72 50.33 54.18 73.80 44.21 39.52 60.23 60.28 58.51
CPM (Li2018) 73.41 65.16 58.31 47.57 74.41 71.13 46.23 41.15 67.97 61.68 60.70
DeepFlow (Weinzaepfel2013) 83.84 81.90 63.33 55.52 83.38 80.87 57.08 56.65 71.25 64.67 69.85
RLOF6 (Geistert2016) 82.80 78.31 63.16 57.68 87.46 86.76 50.56 50.53 69.86 68.73 69.59
RLOF10 (Geistert2016) 80.14 73.95 62.05 55.54 85.44 84.39 48.80 47.84 67.53 67.41 67.31
DIS4 (Kroeger2016) 80.44 76.19 64.11 56.99 82.89 82.24 53.91 52.75 72.11 70.71 69.23
DIS2 (Kroeger2016) 47.55 33.03 36.52 25.32 22.59 19.76 26.79 20.89 27.63 27.91 28.80
Farneback (Farneback2003) 78.69 74.24 65.22 59.43 86.89 87.17 52.85 55.29 70.22 68.94 69.89
Sparse to Dense PLK (Bouguet2000) 75.15 68.54 64.71 57.88 84.71 84.11 50.08 49.26 68.45 69.75 67.26

Person Trajectories

IM01 (Dyn) IM02 (Dyn) IM03 (Dyn) IM04 (Dyn) IM05 (Dyn) Avg.
FlowFields (Bailer2015) 77.94 62.68 52.35 38.22 66.76 63.17 30.09 25.24 65.67 68.20 55.03
RIC (Hu2017) 87.88 80.87 56.56 48.14 43.49 70.98 32.48 27.81 57.47 68.56 57.42
CPM (Li2018) 82.17 68.82 54.56 40.99 70.37 66.69 35.98 30.00 69.64 71.58 59.08
DeepFlow (Weinzaepfel2013) 99.19 95.32 68.60 63.04 83.18 81.20 53.82 52.22 76.32 79.15 75.20
RLOF6 (Geistert2016) 97.70 92.37 66.70 65.08 88.73 90.22 43.56 46.47 72.60 80.12 74.36
RLOF10 (Geistert2016) 96.00 85.02 63.08 59.77 85.97 86.69 39.41 40.48 69.09 78.70 70.42
DIS4 (Kroeger2016) 92.22 85.98 63.97 56.35 81.59 81.61 44.58 42.64 74.95 82.09 70.60
DIS2 (Kroeger2016) 40.81 22.39 22.86 15.37 9.05 6.72 13.63 9.72 17.86 18.10 17.65
Farneback (Farneback2003) 88.75 81.33 64.69 59.05 85.92 87.44 42.42 45.35 71.51 79.63 70.61
Sparse to Dense PLK (Bouguet2000) 79.31 66.83 61.05 52.41 82.63 83.11 37.92 36.81 67.53 76.18 64.38
NMC (IDREES2014) 96.96 90.33 72.18 71.44 92.28 20.70 32.72 42.38 60.15 56.02 63.52

References - Optical Flow Algorithm

@inproceedings
{Bailer2015,
  title = {Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation},
  author={Bailer, C. and Taetz, B. and Stricker, D.},
  booktitle = {International Conference on Computer Vision},
  pages={4015--4023},
  year = {2015}
}
@inproceedings{Hu2017,
  title={Robust interpolation of correspondences for large displacement optical flow},
  author={Hu, Y. and Li, Y. and Song, R.},
  booktitle={Conference on Computer Vision and Pattern Recognition},
  pages={4791--4799},
  year={2017},
}
@article{Li2018, 
  author={Y. Li and Y. Hu and R. Song and P. Rao and Y. Wang}, 
  journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
  title={Coarse-to-Fine PatchMatch for Dense Correspondence}, 
  year={2018}, 
  volume={28}, 
  number={9}, 
  pages={2233-2245}, 
}
@inproceedings{Weinzaepfel2013,
  AUTHOR = {Weinzaepfel, Philippe and Revaud, Jerome and Harchaoui, Zaid and Schmid, Cordelia},
  TITLE = {{DeepFlow: Large displacement optical flow with deep matching}},
  BOOKTITLE = {{Intenational Conference on Computer Vision }},
  YEAR = {2013},
}
@inproceedings{Geistert2016,
	AUTHOR = {Jonas Geistert and Tobias Senst and Thomas Sikora},
	TITLE = {Robust Local Optical Flow: Dense Motion Vector Field Interpolation},
	BOOKTITLE = {Picture Coding Symposium},
	YEAR = {2016},
	PAGES = {1--5},
}
@inproceedings{Kroeger2016, 
  Author = {Till Kroeger and Radu Timofte and Dengxin Dai and Luc Van Gool}, 
  Title = {Fast Optical Flow using Dense Inverse Search}, 
  Booktitle = {European Conference on Computer Vision }, 
  Year = {2016}
 }
@inproceedings{Farneback2003,
  Author = 	 {Gunnar Farneb{\"a}ck},
  Title = 	 {Two-Frame Motion Estimation Based on Polynomial Expansion},
  Booktitle = 	 {Proceedings of the 13th Scandinavian Conference on Image Analysis},
  Pages = 	 {363--370},
  Year = 	 {2003},
 }
@TECHREPORT{Bouguet2000,
  author = {J.-Y. Bouguet},
  title = {Pyramidal Implementation of the Lucas Kanade Feature Tracker},
  institution = {Intel Corporation Microprocessor Research Lab},
  year = {2000},
  type = {Technical {R}eport},
  publisher = {Intel Corporation Microprocessor Research Labs},
  timestamp = {2013.04.03}
}

References - Person Tracking Algorithm

@article{IDREES2014,
 title = "Tracking in dense crowds using prominence and neighborhood motion concurrence",
 journal = "Image and Vision Computing",
 volume = "32",
 number = "1",
 pages = "14 - 26",
 year = "2014",
 author = "Haroon Idrees and Nolan Warner and Mubarak Shah",
}