Light field from stereo
by Brian Chao
Code for ICIP 2021 and ISMAR/TVCG 2022 paper.
Datasets
Datasets are saved on server 140.112.21.58
, in /mnt/data2/bchao/lf/tcsvt_datasets/
:
hci/dataset.h5
inria_dlfd/dataset.h5
inria_lytro/dataset.h5
There are a total of 3 datasets. The original dataset sources are:
The functions for dataset preprocessing is in preprocessing/process_dataset.py
. The light field subviews are organized into a dataset.h5
file for each dataset. You can modify the dataset paths in the preprocessing/config.yaml
config file.
The dataset paths are defined in in function get_dataset_and_loader()
in train_best.py
. The dataset class definition is in lf_datasets.py
. All light field dataset classes, namely:
HCIDataset
(hci)INRIADataset
(inria_lytro)INRIA_DLFD_Dataset
(inria_dlfd)
inherit the LFDataset
class. The arguments are:
root
: Path todataset.h5
filetrain
: Training or testing datim_size
: Light field sub-view image size for training or testing. Set to 135 in all training experiments. Set to full image resolution (usually 512) when testing.transform
: Preprocessing function to apply on input light field datause_crop
: If specified, crop image toim_size
. Otherwise, resize image toim_size
.mode
: Output data format.stereo_wide
orstereo_narrow
: used in proposed method. For_wide
, the stereo images extracted from the light field is left and right most subviews from the middle row of the whole light field subview grid. For_narrow
, the stereo images are the left and right images adjacent to the center view. See ISMAR/TVCG 2022 paper for more info.single
: used in method proposed by Srinivasan et. al. The center view is returned.4crop
,2crop_wide
,2crop_narrow
: used in method proposed by Kalantari et. al. The stereo images returned in modes2crop_wide
and2crop_narrow
are the same as that ofstereo_wide
andstereo_narrow
, respectively.4crop
returns the 4 corner views in the light field. Note that the target image in this mode is only a single subview is returned in this mode, unlike whole light field as instereo_wide
orstereo_narrow
.
Trained weights
Trained weights are saved on server 140.112.21.58
, in /mnt/data2/bchao/tvcg_models
, organized by dataset name and experiment name. The directory tree structure is as follows:
/mnt/data2/bchao/tvcg_models
hci
20211126_1420_final
ckpt
disp_1000.ckpt
refine_1000.ckpt
- ...
inria_dlfd
- ...
Training and testing
Main training script: train_best.py
. Testing script is test.py
.
See bash_run_scripts/train.sh
and bash_run_scripts/test.sh
for more details on how to use arguments.
Training arguments
imsize
: Image size of dataset.save_epochs
: Save model frequency.use_crop
: Crop or resize image toimsize
. If specified, use cropping rather than resize.max_disparity
: disparity scaling factor for input stereo images. Value depends on datasets.
Configuration | max_disparity value |
---|---|
hci | 32 |
inria_dlfd | 40 |
inria_lyro | 10 |
disp_model
: Disparity network architecture. Defaultoriginal
.refine_model
: Refinement network architecture. Default3dcnn
.refine_hidden
: Refinement network hidden channels. Default 128.merge_method
: Merge method (see paper for more details). Best model usesalpha
.recon_loss
: Reconstruction loss. Defaultl1
.consistency_w
: Weight of left-right consistency loss.tv_loss_w
: Weight of disparity smoothing loss.dataset
: Light field dataset to use. Currently supported:hci
,inria_lytro
,inria_dlfd
.name
: Name of experiment.mode
: input stereo image configurationstereo_wide
orstereo_narrow
. See paper or above for more details on this argument.
Additional testing arguments
test_mode
: Testing modenormal
: Compute PSNR, SSIM over all testing datahorizontal
: Computer PSNR, SSIM over middle row subviews in light field testing data. To compare with Zhang et. al.stereo
: Use own stereo images as testing data.
OLD: For software team (Eric and Andy)
Data and weights
- Extract
production.tgz
- Find model weights (.ckpt files) in extracted
production/weights/hci
,production/weights/inria
folder - Put
disp_2000.ckpt
andrefine_2000.ckpt
model weights in:experiments/hci/deploy/ckpt
for HCI modelexperiments/inria/deploy/ckpt
for INRIA model
- Find testing data in
production/testing_data/hci_testing_data
,production/testing_data/inria_testing_data
folder- Stereo images are labelled by their id (0, 1, 2, ...) and
left
,right
prefixes - During inference, put stereo images in
data/
folder - Modify file path in
single_stereo
function intest.py
- Stereo images are labelled by their id (0, 1, 2, ...) and
Inference scripts
- See
bash_scripts_software/