/Video-Classification-2-Stream-CNN

Video Classification using 2 stream CNN

Primary LanguagePython

Video Classification using Two Stream CNNs

We use a spatial and a temporal stream with VGG-16 and CNN-M respectively for modeling video information. LSTMs are stacked on top of the CNNs for modeling long term dependencies between video frames. For more information, see these papers:

Two-Stream Convolutional Networks for Action Recognition in Videos

Fusing Multi-Stream Deep Networks for Video Classification

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Towards Good Practices for Very Deep Two-Stream ConvNets


Here are the steps to run the project on CCV dataset:

Creating a virtual environment

First create a directory named env and then run the following inside the directory. This will create a virtual environment. Assuming we create a requirements.txt file to help install modules that are needed in the project. $ mkdir env $ cd env $ virtualenv venv-video-classification $ source env-video-classification\bin\activate $ cd .. $ pip install requirements.txt

Setting up the DataSet

  1. Get the YouTube data, remove broken videos and negative instances and finally create a pickle file of the dataset by running scripts from the utility_scripts folder

  2. Temporal Stream (in the temporal folder):

  3. Run temporal_vid2img to create optical flow frames and the related files

  4. Run temporal_stream_cnn to start with the temporal stream training

  5. Spatial Stream (in the spatial folder):

  6. Run the spatial_vid2img to create static frames and related files

  7. Download the vgg16_weights.h5 file from here and put it in the spatial folder

  8. Run spatial_stream_cnn to start with the spatial stream training

  9. Temporal Stream LSTM: Will soon update the code

  10. Spatial Stream LSTM: Will soon update the code