This project is based on the paper Towards Semantic Fast-Forward and Stabilized Egocentric Videos on the First International Workshop on Egocentric Perception, Interaction and Computing at European Conference on Computer Vision Workshops (EPIC@ECCVW 2016). It implements a semantic fast-forward method for First-Person videos with a proper stabilization method.
For more information and visual results, please access the project page.
- Michel Melo da Silva - PhD student - UFMG - michelms@dcc.ufmg.com
- Washington Luis de Souza Ramos - MSc student - UFMG - washington.ramos@outlook.com
- João Pedro Klock Ferreira - Undergraduate Student - UFMG - jpklock@ufmg.br
- Mario Fernando Montenegro Campos - Advisor - UFMG - mario@dcc.ufmg.br
- Erickson Rangel do Nascimento - Advisor - UFMG - erickson@dcc.ufmg.br
Federal University of Minas Gerais (UFMG)
Computer Science Department
Belo Horizonte - Minas Gerais -Brazil
VeRLab: Laboratory of Computer Vison and Robotics
http://www.verlab.dcc.ufmg.br
This project is a two-fold source code. The first fold¹ is composed of MATLAB code to describe the video semantically and to fast-forward it. A stabilizer proper to fast-forwarded video written in C++ using OpenCV is the second fold². You can run each fold separately.
- ¹MATLAB 2015a or higher
- ²OpenCV 2.4 (Tested with 2.4.9 and 2.4.13)
- ²Armadillo 6 (Tested with 6.600.5 -- Catabolic Amalgamator)
- ²Boost 1 (Tested with 1.54.0 and 1.58.0)
- ²Doxygen 1 (for documentation only - Tested with 1.8.12)
The project processing is decribed by the following flowchart:
-
Optical Flow Estimator:
The first step processing is to estimate the Optical Flow of the Input VIdeo.
- First, you should download the Poleg et al. 2014 Flow Estimator code from the link.
- Navigate to the download folder and unzip the code.
- Into the Vid2OpticalFlowCSV\Example folder, run the command:
Vid2OpticalFlowCSV.exe -v < video_filename > -c < config.xml > -o < output_filename.csv >| Options | Description | Type | Example |
|---|---|---|---|
< Video_filename > |
Path and filename of the video. | String | ~/Data/MyVideos/myVideo.mp4 |
< config.xml > |
Path to the configuration XML file. | String | ../default-config.xml |
< output_filename.csv > |
Path to save the output CSV file. | String | myVideo.csv |
Save the output file using the same name of the input video with extension .csv.
-
Semantic Extractor:
The second step is to extract the semantic information over all frames of the Input video and save it in a CSV file. On the MATLAB console, go to the project folder and run the command:
>> ExtractAndSave(< Video_filename >, < Semantic_extractor_name >)
| Parameters | Description | Type | Example |
|---|---|---|---|
< Video_filename > |
Path and filename of the video. | String | ~/Data/MyVideos/Video.mp4 |
< Semantic_extractor_name > |
Semantic extractor algorithm. | String | 'face' or 'pedestrian' |
-
Calculate Speed-up rates
To calculate the speed-up rates for each type of segment, on the MATLAB console, go to the project folder and run the following commands:
>> [~, < Num_non_semantic_Frames >, < Num_semantic_frames >] = GetSemanticRanges(< Semantic_Data_MAT_filename >);| Parameters | Description | Type | Example |
|---|---|---|---|
< Semantic_Data_MAT_filename > |
File created by the step 2 | String | '../example_face_extracted.mat' |
| Output | Description | Type | Example |
|---|---|---|---|
< Num_non_semantic_Frames > |
Number of frames in the Non-Semantic segments. (Will be used below.) | Integer | Tns |
< Num_semantic_Frames > |
Number of frames in the Semantic segments. (Will be used below.) | Integer | Ts |
>> SpeedupOptimization( < Num_non_semantic_Frames >, < Num_semantic_frames >, < Desired_speedup >, < Max_speedup >, < lambda_1 >, < lambda_2 >, < show_plot > )| Parameters | Description | Type | Example |
|---|---|---|---|
< Num_non_semantic_Frames > |
Number of frames in the Non-Semantic segments. (Obtained from the previous code) | Integer | Tns |
< Num_semantic_Frames > |
Number of frames in the Semantic segments. (Obtained from the previous code) | Integer | Ts |
< Desired_speedup > |
Desired speed-up rate to the whole video. | Integer | 10 |
< Max_speedup > |
Maximum allowed jump. | Integer | 100 |
< Lambda_1 > |
Value of Lambda 1 in the optimization function. | Integer | 40 |
< Lambda_2 > |
Value of Lambda 2 in the optimization function. | Integer | 8 |
< show_plot > |
Flag to show the search space create by the optimization function. | Boolean | false |
-
Create Experiment
To run the code, you should create an experiment entry. On a text editor, add a case to the
GetVideoDetailsfunction in the fileSemanticSequenceLibrary.m:
function [videoFile, startInd, endInd, filename, fps] = GetVideoDetails(video_dir,exp_name)
...
case < Experiment_name >
filename = < video_filename >;
startInd = < start_index_frame > ;
endInd = < final_index_frame >;
fps = < video_frames_per_second >;
...
| Fields | Description | Type | Example |
|---|---|---|---|
< Experiment_name > |
Name to identify the experiment. | String | MyVideo |
< video_filename > |
Filename to the video. | String | myVideo.mp4 |
< start_index_frame > |
Frame index to start the processing. | Integer | 1 |
< final_index_frame > |
Frame intex to stop the processing. | Integer | 16987 |
< video_frames_per_second > |
Frames per second of the video. | Integer | 30 |
-
Semantic Fast-Forward
After the previous steps, you are ready to accelerate the Input Video. On MATLAB console, go to the project folder and run the command:
>> SpeedupVideo(< Video_dir >, < Experiment_name >, < Semantic_extractor_name >, < Semantic_speedup >, < Non_semantic_speedup >, < Shakiness_weights >, < Velocity_weights >, < Appearance_weights >, < Semantic_weights > )| Parameters | Description | Type | Example |
|---|---|---|---|
< Video_dir > |
Path to the folder where the video file is. | String | ~/Data/MyVideos |
< Experiment_name > |
Name set in the SemanticSequenceLibrary.m file. | String | My_Video |
< Semantic_extractor_name > |
Semantic extractor algorithm used in the Semantic Extractor step. | String | 'face' or 'pedestrian' |
< Semantic_speedup > |
Desired speed-up assign to the semantic segments of the input video. | Integer | 3 |
< Non_semantic_speedup > |
Desired speed-up assign to the semantic segments of the input video. | Integer | 14 |
< Shakiness_weights > |
Tuple of weights related to the shakiness term in the edge weight formulation. | [Integer, Integer] | [10,250] |
< Velocity_weights > |
Tuple of weights related to the shakiness term in the edge weight formulation. | [Integer, Integer] | [50,3000] |
< Appearance_weights > |
Tuple of weights related to the shakiness term in the edge weight formulation. | [Integer, Integer] | [0,100] |
< Semantic_weights > |
Tuple of weights related to the shakiness term in the edge weight formulation. | [Integer, Integer] | [500,8] |
- Configure Video Parameters
After the Semantic Fast-Forward step, the accelerated video is create. Now we are going to stabilize the output video. The first stabilization step is to configure the video parameters in the file acceleratedVideoStabilizer/experiment.xml. Follow the instructions described into the file.
-
Accelerate Video Stabilizer
Navigate to the
<project_folder>/acceleratedVideoStabilizer/folder. Follow the instructions described into the<project_folder>/acceleratedVideoStabilizer/README.mdfile to compile and run the code.
The output of this step is the stabilized semantic fast-forward video.
- EgoSampling: Y. Poleg, T. Halperin, C. Arora, S. Peleg, Egosampling: Fast-forward and stereo for egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4768–4776. doi:10.1109/CVPR.2015.7299109.
- LK_Blocked_Optical_Flow: Y. Poleg, C. Arora, S. Peleg, Temporal segmentation of egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2537–2544. doi:10.1109/CVPR.2014.325.
- NPD_Face_Detector: S. Liao, A. K. Jain, S. Z. Li, A fast and accurate unconstrained face detector, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2) (2016) 211–223. doi:10.1109/TPAMI.2015.2448075.
- PMT_Pedestrian_Detector: P. Dollar, Piotr’s Computer Vision Matlab Toolbox (PMT), https://github.com/pdollar/toolbox.
If you are using it for academic purposes, please cite:
M. M. Silva, W. L. S. Ramos, J. P. K. Ferreira, M. F. M. Campos, E. R. Nascimento, Towards semantic fast-forward and stabilized egocentric videos, in: European Conference on Computer Vision Workshops, Springer International Publishing, Amsterdam, NL, 2016, pp. 557–571. doi:10.1007/978-3-319-46604-0_40.
@InBook{Silva2016,
Title = {Towards Semantic Fast-Forward and Stabilized Egocentric Videos},
Author = {Silva, Michel Melo and Ramos, Washington Luis Souza and Ferreira, Joao Pedro Klock and Campos, Mario Fernando Montenegro and Nascimento, Erickson Rangel},
Editor = {Hua, Gang and J{'e}gou, Herv{'e}},
Pages = {557--571},
Publisher = {Springer International Publishing},
Year = {2016},
Address = {Cham},
Booktitle = {Computer Vision -- ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part I},
Doi = {10.1007/978-3-319-46604-0_40},
ISBN = {978-3-319-46604-0},
Url = { http://dx.doi.org/10.1007/978-3-319-46604-0_40 }
}