Video compressive sensing aims at increasing the temporal resolution of a sensor by incorporating additional hardware components to the camera architectuand employing powerful computational techniques for high speed video reconstruction. The additional components operate at higher frame rates than the camera’s native temporal resolution giving rise to low frame rate multiplexed measurements (frames in the middle) which can later be decoded to extract the unknown observed high speed video sequence (frames in the right).
DeepVideoCS
A deep learning framework for video compressive sensing [Paper] [Project Page]
Deep Fully-Connected Networks for Video Compressive Sensing Michael Iliadis*, Leonidas Spinoulas*, Aggelos K. Katsaggelos In Elsevier Digital Signal Processing, 2018. (* equal contributions)
Prerequisites
- Linux or MacOS
- Python 2
- NVIDIA GPU + CUDA CuDNN
Getting Started
Installation
- Install PyTorch and dependencies from http://pytorch.org
Python requirements
Install all requirements below (with pip or conda)
- imageio (https://imageio.github.io/)
- h5py (http://www.h5py.org/)
- pandas (https://pandas.pydata.org/)
- bokeh (https://bokeh.pydata.org/en/latest/)
Clone this repo
git clone https://github.com/miliadis/DeepVideoCS
cd DeepVideoCS
Testing
Download testing videos
- paper_videos: 14 Videos that we tested in our paper
Download pre-trained models
Download the pre-trained models using the following script:
bash ./download/download_model.sh model_name
fc7
: Fully connected network with 7 layersfc4
: Fully connected network with 4 layers
Testing script
To reproduce paper's results please run the following script:
python test.py './download/papers_video' fcnet 7 './download/model_best.pth.tar' --save_videos='./output_videos' --gpu_id 0
If you want to reproduce results with noise use the option 'noise'
.
Here are all options:
usage: test.py [-h] [--block_opts BLOCK_OPTS [BLOCK_OPTS ...]]
[--block_overlap] [--noise NOISE] [--mean MEAN] [--std STD]
[--seed SEED] [--save_videos SAVE_VIDEOS]
[--save_format SAVE_FORMAT] [--gpu_id GPU_ID]
data arch layers_k pretrained_net
PyTorch Video Compressive Sensing - Testing
positional arguments:
data path to testing dataset
arch choose model name
layers_k number of FC layers in decoder
pretrained_net pre-trained model path
optional arguments:
-h, --help show this help message and exit
--block_opts BLOCK_OPTS [BLOCK_OPTS ...]
Item order: (temporal size, spatial size, video
chunks)
--block_overlap overlapping blocks or not)
--noise NOISE Noise Level in dB: e.g., 20, 30, 40
--mean MEAN Mean file
--std STD Standard deviation file
--seed SEED random seed
--save_videos SAVE_VIDEOS
path to save videos
--save_format SAVE_FORMAT
format for saving video file: avi, gif
--gpu_id GPU_ID choose gpu id
Citation
If you use this code for your research, please cite our papers.
@article{DeepVideoCS2018,
title = "Deep fully-connected networks for video compressive sensing",
journal = "Digital Signal Processing",
volume = "72",
pages = "9 - 18",
year = "2018",
url = "http://www.sciencedirect.com/science/article/pii/S1051200417302130",
author = "Michael Iliadis and Leonidas Spinoulas and Aggelos K. Katsaggelos"}
Acknowledgments
Code is inspired by pytorch-imagenet-example, convnets-in-pytorch and cyclegan.