A PyTorch implementation of PredRNN [paper], a recurrent network with twisted and zigzag space-time memory cells for video data. Given a sequence of previous frames, our model generates future frames for multiple timestamps.
Video prediction networks have been used for precipitation nowcasting, early activity recognition, physical scene understanding, model-based visual planning, and unsupervised representation learning of video data.
-
Install Python 3.7, PyTorch 1.3, and OpenCV 3.4.
-
Download data. This repo contains code for two datasets: the Moving Mnist dataset and the KTH action dataset.
-
Train the model. You can use the following bash script to train the model. The learned model will be saved in the
--save_dirfolder. The generated future frames will be saved in the--gen_frm_dirfolder.
cd script/
sh predrnn_mnist_train.sh
If you use this repo or our results in your research, please remember to cite the following paper.
@inproceedings{wang2017predrnn,
title={Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms},
author={Wang, Yunbo and Long, Mingsheng and Wang, Jianmin and Gao, Zhifeng and Philip, S Yu},
booktitle={Advances in Neural Information Processing Systems},
pages={879--888},
year={2017}
}
PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning.
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S. Yu.
ICML 2018 [paper] [code]
You may send email to yunbo.thu@gmail.com or longmingsheng@gmail.com, or create an issue in this repo and @wyb15.