CS166 Final Project: Event Camera Video Reconstruction

Running with Anaconda

cuda_version=11.8

conda create -y -n firenet python=3.9
conda activate firenet
conda install -y pytorch torchvision cudatoolkit=$cuda_version -c pytorch
pip install -r requirements.txt

Inference

Download the pretrained models from here.

You can pick any event videos from DAVIS and HDR. If you download from DAVIS, specify the loader type as Davis. If you download from HDR, specify the loader type as HDR. Make sure to put the events and video/images in the same folder to assure the inference works.

To estimate reconstruction:

python inference.py --checkpoint_path <path/to/model.pth> --events_file_path </path/to/events> --loader_type <data tpye> --output_folder </path/to/output/dir>

For example:

python inference.py \
    --checkpoint_path ./checkpoints/checkpoint-epoch970.pth \
    --events_file_path ./Davis/slider_depth \
    --loader_type Davis \
    --output_folder results/inference/slider_depth

Training dataset

You will need to generate the training dataset yourself, using ESIM. To find out how, please see the training data generator repo.

Or you can download the dataset here.

Training

To train a model, you need to create a config file (see config/config.json for an example). In this file, you need to set what model you would like to use, but I only make FireNet work. You also need to set the training parameters, the training data, the validation data and the output directory. You can then start the training by invoking

python train.py --config path/to/config

If you have a model that would like to keep training from, you can use

python train.py --config path/to/config --resume /path/to/model.pth