/SemanticFastForward_ECCVW_2016

Towards Semantic Fast-Forward and Stabilized Egocentric Videos @ 2016 First International Workshop on Egocentric Perception, Interaction and Computing at European Conference on Computer Vision Workshops (EPIC@ECCVW)

Primary LanguageMATLABGNU General Public License v3.0GPL-3.0

Version License

Project

This project is based on the paper Towards Semantic Fast-Forward and Stabilized Egocentric Videos on the First International Workshop on Egocentric Perception, Interaction and Computing at European Conference on Computer Vision Workshops (EPIC@ECCVW 2016). It implements a semantic fast-forward method for First-Person videos with a proper stabilization method.

For more information and visual results, please access the project page.

Contact

Authors

Institution

Federal University of Minas Gerais (UFMG)
Computer Science Department
Belo Horizonte - Minas Gerais -Brazil

Laboratory

VeRLab

VeRLab: Laboratory of Computer Vison and Robotics
http://www.verlab.dcc.ufmg.br

Code

This project is a two-fold source code. The first fold¹ is composed of MATLAB code to describe the video semantically and to fast-forward it. A stabilizer proper to fast-forwarded video written in C++ using OpenCV is the second fold². You can run each fold separately.

Dependencies

  • ¹MATLAB 2015a or higher
  • ²OpenCV 2.4 (Tested with 2.4.9 and 2.4.13)
  • ²Armadillo 6 (Tested with 6.600.5 -- Catabolic Amalgamator)
  • ²Boost 1 (Tested with 1.54.0 and 1.58.0)
  • ²Doxygen 1 (for documentation only - Tested with 1.8.12)

Usage

The project processing is decribed by the following flowchart:

Flowchart

  1. Optical Flow Estimator:

    The first step processing is to estimate the Optical Flow of the Input VIdeo.

    1. First, you should download the Poleg et al. 2014 Flow Estimator code from the link.
    2. Navigate to the download folder and unzip the code.
    3. Into the Vid2OpticalFlowCSV\Example folder, run the command:
Vid2OpticalFlowCSV.exe -v < video_filename > -c < config.xml > -o < output_filename.csv >
Options Description Type Example
< Video_filename > Path and filename of the video. String ~/Data/MyVideos/myVideo.mp4
< config.xml > Path to the configuration XML file. String ../default-config.xml
< output_filename.csv > Path to save the output CSV file. String myVideo.csv

Save the output file using the same name of the input video with extension .csv.

  1. Semantic Extractor:

    The second step is to extract the semantic information over all frames of the Input video and save it in a CSV file. On the MATLAB console, go to the project folder and run the command:

>> ExtractAndSave(< Video_filename >, < Semantic_extractor_name >)
Parameters Description Type Example
< Video_filename > Path and filename of the video. String ~/Data/MyVideos/Video.mp4
< Semantic_extractor_name > Semantic extractor algorithm. String 'face' or 'pedestrian'
  1. Calculate Speed-up rates

    To calculate the speed-up rates for each type of segment, on the MATLAB console, go to the project folder and run the following commands:

>> [~, < Num_non_semantic_Frames >, < Num_semantic_frames >] = GetSemanticRanges(< Semantic_Data_MAT_filename >);
Parameters Description Type Example
< Semantic_Data_MAT_filename > File created by the step 2 String '../example_face_extracted.mat'
Output Description Type Example
< Num_non_semantic_Frames > Number of frames in the Non-Semantic segments. (Will be used below.) Integer Tns
< Num_semantic_Frames > Number of frames in the Semantic segments. (Will be used below.) Integer Ts
>> SpeedupOptimization( < Num_non_semantic_Frames >, < Num_semantic_frames >, < Desired_speedup >, < Max_speedup >, < lambda_1 >, < lambda_2 >, < show_plot > )
Parameters Description Type Example
< Num_non_semantic_Frames > Number of frames in the Non-Semantic segments. (Obtained from the previous code) Integer Tns
< Num_semantic_Frames > Number of frames in the Semantic segments. (Obtained from the previous code) Integer Ts
< Desired_speedup > Desired speed-up rate to the whole video. Integer 10
< Max_speedup > Maximum allowed jump. Integer 100
< Lambda_1 > Value of Lambda 1 in the optimization function. Integer 40
< Lambda_2 > Value of Lambda 2 in the optimization function. Integer 8
< show_plot > Flag to show the search space create by the optimization function. Boolean false
  1. Create Experiment

    To run the code, you should create an experiment entry. On a text editor, add a case to the GetVideoDetails function in the file SemanticSequenceLibrary.m:

function [videoFile, startInd, endInd, filename, fps] = GetVideoDetails(video_dir,exp_name)

 ...

 case < Experiment_name >
     filename = < video_filename >;
     startInd = < start_index_frame > ;
     endInd   = < final_index_frame >;
     fps      = < video_frames_per_second >;
					
  ... 
  
Fields Description Type Example
< Experiment_name > Name to identify the experiment. String MyVideo
< video_filename > Filename to the video. String myVideo.mp4
< start_index_frame > Frame index to start the processing. Integer 1
< final_index_frame > Frame intex to stop the processing. Integer 16987
< video_frames_per_second > Frames per second of the video. Integer 30
  1. Semantic Fast-Forward

    After the previous steps, you are ready to accelerate the Input Video. On MATLAB console, go to the project folder and run the command:

>> SpeedupVideo(< Video_dir >, < Experiment_name >, < Semantic_extractor_name >, < Semantic_speedup >, < Non_semantic_speedup >, < Shakiness_weights >, < Velocity_weights >, < Appearance_weights >, < Semantic_weights > )
Parameters Description Type Example
< Video_dir > Path to the folder where the video file is. String ~/Data/MyVideos
< Experiment_name > Name set in the SemanticSequenceLibrary.m file. String My_Video
< Semantic_extractor_name > Semantic extractor algorithm used in the Semantic Extractor step. String 'face' or 'pedestrian'
< Semantic_speedup > Desired speed-up assign to the semantic segments of the input video. Integer 3
< Non_semantic_speedup > Desired speed-up assign to the semantic segments of the input video. Integer 14
< Shakiness_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [10,250]
< Velocity_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [50,3000]
< Appearance_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [0,100]
< Semantic_weights > Tuple of weights related to the shakiness term in the edge weight formulation. [Integer, Integer] [500,8]
  1. Configure Video Parameters

  After the Semantic Fast-Forward step, the accelerated video is create. Now we are going to stabilize the output video. The first stabilization step is to configure the video parameters in the file acceleratedVideoStabilizer/experiment.xml. Follow the instructions described into the file.

  1. Accelerate Video Stabilizer

    Navigate to the <project_folder>/acceleratedVideoStabilizer/ folder. Follow the instructions described into the <project_folder>/acceleratedVideoStabilizer/README.md file to compile and run the code.

The output of this step is the stabilized semantic fast-forward video.

Acknowledgements

  1. EgoSampling: Y. Poleg, T. Halperin, C. Arora, S. Peleg, Egosampling: Fast-forward and stereo for egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4768–4776. doi:10.1109/CVPR.2015.7299109.
  2. LK_Blocked_Optical_Flow: Y. Poleg, C. Arora, S. Peleg, Temporal segmentation of egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2537–2544. doi:10.1109/CVPR.2014.325.
  3. NPD_Face_Detector: S. Liao, A. K. Jain, S. Z. Li, A fast and accurate unconstrained face detector, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2) (2016) 211–223. doi:10.1109/TPAMI.2015.2448075.
  4. PMT_Pedestrian_Detector: P. Dollar, Piotr’s Computer Vision Matlab Toolbox (PMT), https://github.com/pdollar/toolbox.

Citation

If you are using it for academic purposes, please cite:

M. M. Silva, W. L. S. Ramos, J. P. K. Ferreira, M. F. M. Campos, E. R. Nascimento, Towards semantic fast-forward and stabilized egocentric videos, in: European Conference on Computer Vision Workshops, Springer International Publishing, Amsterdam, NL, 2016, pp. 557–571. doi:10.1007/978-3-319-46604-0_40.

Bibtex entry

@InBook{Silva2016,
Title = {Towards Semantic Fast-Forward and Stabilized Egocentric Videos},
Author = {Silva, Michel Melo and Ramos, Washington Luis Souza and Ferreira, Joao Pedro Klock and Campos, Mario Fernando Montenegro and Nascimento, Erickson Rangel},
Editor = {Hua, Gang and J{'e}gou, Herv{'e}},
Pages = {557--571},
Publisher = {Springer International Publishing},
Year = {2016},
Address = {Cham},
Booktitle = {Computer Vision -- ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part I},
Doi = {10.1007/978-3-319-46604-0_40},
ISBN = {978-3-319-46604-0},
Url = { http://dx.doi.org/10.1007/978-3-319-46604-0_40 }
}

Enjoy it.