MMVP: Motion-Matrix based Video Prediction

arXiv

git clone https://github.com/Kay1794/MMVP-motion-matrix-based-video-prediction.git
cd MMVP-motion-matrix-based-video-prediction
conda env create -f environment.yml

Data Preparation

MMVP has been tested on three datasets: UCF Sports, KTH and Moving MNIST

Please feel free to download our pre-processed dataset:

UCF Sports MMVP split;

UCF Sports STRPM split (preprocessed data link TBD, please first refer to STIP repo);

KTH;

Moving MNIST.

The final data folder PATH_TO_DATASET should be organized like this:

PATH_TO_DATASET
├── ucf_ours
│   ├──ucf_prediction_4to1_filter_easy_test  
│   ├──ucf_prediction_4to1_filter_intermediate_test  
│   ├──ucf_prediction_4to1_filter_train
│   ├──ucf_prediction_4to1_filter_hard_test  
│   └──ucf_prediction_4to1_filter_test 
│   │   ├──XXX.npy
│   │   └──...
├── kth
│   ├──test_indices_gzip.hkl       
│   ├──train_data_gzip_t=40.hkl
│   ├──test_data_gzip.hkl       
│   ├──test_indices_gzip_t=40.hkl  
│   ├──train_indices_gzip.hkl
│   ├──test_data_gzip_t=40.hkl  
│   ├──train_data_gzip.hkl         
│   └──train_indices_gzip_t=40.hkl
├── moving-mnist
│   ├── mnist_test_seq.npy  
│   └── train-images-idx3-ubyte.gz
├── ucf_strpm
    └── ...

Please modify PATH_TO_DATASET in config.py to your own dataset root.

Run

Please add --use_direct_predictor for kth and moving-mnist dataset

Training

python main.py --exp EXP_NAME --base_channel 16 --pred_base_channel 8 --mode train --dataset ucf_4to1 --rot_aug --flip_aug --cos_restart --t_period 50 --nepoch 50 --loss_list recon --batch 3 --logpath ./results/ --filter_block --scale_in_use 3 --res_cat_img --downsample_scale 2 4 2 --lr 1e-3 --log

Validation

python main.py --exp EXP_NAME --base_channel 16 --pred_base_channel 8 --mode val --dataset ucf_4to1 --rot_aug --flip_aug --cos_restart --t_period 50 --nepoch 50 --loss_list recon --batch 3 --logpath ./results/ --filter_block --scale_in_use 3 --res_cat_img --downsample_scale 2 4 2 --lr 1e-3 --log --resume PATH_TO_PTH_FILE