IVFuseNet: Fusion of Infrared and Visible Images for Depth Prediction

This repository includes our model code and dataset for depth prediction task.

This code works on Python 3 & TensorFlow 1.4 and the images in this dataset are of actual road scenes captured while unmanned vehicle driving.

If this code and dataset are provided for research purposes, please see License section below.

Model

This model's structure looks just like the following illustration:

Train

python train.py

Test

Jupyter notebook test.ipynb

Data_Preprocess

Our NUST-SR dataset is composed of the actual road scenes captured while unmanned vehicle driving in the daytime and night. Currently the dataset contains visible light images, infrared images and depth map:

                        |daytime| night  
------------------------------------------
infrared images         | 6529  |  5612
visible light images    | 6529  |  5612
raw depth maps          | 6529  |  5612
------------------------------------------

The raw depth map should be preprocessed, firstly, match the visible light images, infrared images and depth map, cut them into 256*512:

jupyter notebook rgb_ir_depth.ipynb

Secondly,filled the points for which there is no depth value first using colorization scheme of Levin et al in the NYUDepth development kit:

jupyter notebook fill_points.ipynb

Finally, shuffle images and generate the dataset （the final depth map are classified into 32 classes）:

jupyter notebook shuffle picture.ipynb

The example of raw depth map, depth map with filled points and the final depth map are shown below:

Here we give the data link , please click the link to get the visible light images, infrared images and the corresponding ground-truth depths:

NUST-SR

usage of NUST-SR

Click the link to get the example of dataset of daytime and night:

Dataset-example

Start with importing package:

import h5py
import matplotlib.pyplot as plt

To load a dataset:

def read_hdf5(file_name):
    with h5py.File(file_name, 'r') as f:
        images = np.asarray(f['images'])
        depths = np.asarray(f['depths'])
        infrareds = np.asarray(f['infrareds'])
    return images,depths,infrareds
images,depths,infrareds=read_hdf5('test_snow_data.h5')

To display the image of a dataset:

i, j = 0,4
imageTest = images[i:j]
plt.imshow(imageTest[0],cmap='jet')

License

I provide this project for research purposes, please follow Citing.

For removal of copyrighted content, please contact me on GitHub.

Citing

If you use this project in academic work, please cite as follows:

@misc{liyuqi1234,
        title={IVFuseNet: Fusion of Infrared and Visible Images for Depth Prediction},
        url={https://github.com/liyuqi1234/IVFN},
        author={Yuqi, Li},
        year={2019},
        publisher = {GitHub},
        journal = {GitHub repository}
}