/rlc

Primary LanguageC++OtherNOASSERTION

=================================================== Important

Raytrix Lenslet content Convertor 2.0 (RLC2.0)

Copyright: ONLY Available for ACADEMIC and MPEG Standardization Purposes Production:

  • Nagoya University
  • Université Libre de Bruxelles

=================================================== Description

Given a set of Raw video captured by multi-focused plenoptic camera (i.e. Raytrix camera,https://raytrix.de/) and its microlens calibration data, the provided source code can render multi-view video.

This tool is implemented by:

Fujii Laboratory Department of Information and Communication Engineering Graduate School of Engineering Nagoya University, Japan

This tool is also updated and maintained by:

Laboratory of Image Synthesis and Analysis - Virtual Reality (LISA-VR) École Polytechnique de Bruxelles Université Libre de Bruxelles, Belgium

=================================================== Term of Use

Term of Use: ANY kind of academic publication, and report using this tool MUST refer to the following TWO references.

[1] Shu Fujita, Sho Mikawa, Mehrdad Teratani, Keita Takahashi, and Toshiaki Fujii “Extracting Multi-View Images from Multi-Focused Plenoptic Camera,” International Workshop on Advanced Image Technology (IWAIT)/International Forum on Medical Imaging in Asia (IFMIA), in SPIE Digital Library, Singapore, January 2019

[2] ] Mehrdad Teratani, Shu Fujita, Wenzhe Ouyang, Keita Takahashi, Toshiaki Fujii, "A 3-D Imaging System Using Multi-Focused Plenoptic Camera and Tensor Display," 2018 International Conference on 3D Immersion (IC3D2018) in IEEE Xplore Digital Library, Brussels, Belgium, December 2018

=================================================== Prerequisites

  1. Boost
  2. OpenCV 4.0 or above
  3. RLC2.0 can be complied on Windows. RLC0.3 was implemented on Ubuntu 18.04 LTS with g++ 7.3.0 compiler.

=================================================== Instruction to run the program

To build the software, execute the make command in the working directory. $ make An executable program, named 'converting' will be generated.

To demonstrate the software, we have included some test data captured by Raytrix R5, you can execute 'RLC' with a parameter file. $ ./converting param.cfg [--memory]

When the program finishes, it will automatically create a folder for storing the multi-view video. Please make sure that you designate valid file paths.

Notes:

  1. It is recommended to do a test with single frame input data firstly, while you set the parameters to default value. If you find that the generated three lens-type images (LensType_1.png, LensType_2.png and LensType_3.png) are not arranged with default mode (lmode: FMN), please switch it to the suitable one. By single frame conversion, you may set the other parameters too.

  2. RLC is designed for video processing, so we added frame number on input filename. In the case that you want to process a single image without a frame number, please append a frame number to the filename.

  3. In the case that input data is YUV420 format, please specify height and width of input data.

  4. RLC 2.0 is successfully tested for conversion of raw video captured by Raytrix R5 to multi-view video.

  5. The option --memory makes RLC running without the precomputation of the upscaled patches, thus requiring less memory but leading to a slower execution (2 to 3 times).

  6. It is recommended to do a test with single frame input data firstly, while you set the parameters to default value. If you find that the generated three lens-type images (LensType_1.png, LensType_2.png and LensType_3.png) are not arranged with default mode (lmode: FMN), please switch it to the suitable one. By single frame conversion, you may set the other parameters too.

  7. RLC is designed for video processing, so we added frame number on input filename. In the case that you want to process a single image without a frame number, please append a frame number to the filename.

  8. In the case that input data is YUV420 format, please specify height and width of input data.

  9. RLC 1.5 is successfully tested for conversion of raw video captured by Raytrix R5 to multi-view video.

=================================================== Parameters

viewNum 5 rmode 1 pmode 0 mmode 2 lmode 1 Calibration_xml .././TestDataset/R5_fujita/CalibData.xml RawImage_Path .././TestDataset/R5_fujita/img%03d.png Output_Path .././Output/fujita/Res_%03d Debayer_mode 0 Isfiltering 0 isCLAHE 0 Gamma 1.0 Lambda 0.05 Sigma 0 input_model 0 output_model 0 start_frame 1 end_frame 2 height 2048 width 2048

=================================================== Parameter Description

viewNum: Number of views at one direction. For example: viewNum = 5 means 5x5 views in 2D configuration Currently, we recommend you to set this value to 5 or 7 to ensure good rendering quality from Lenslet video to Multi-view Video. viewNum = 5 (default value) 5x5 views in 2D configuration viewNum = 7 7x7 views in 2D configuration

rmode: Resolution mode, and this value must be a integer. It is corresponded to the "skipped_number_of_pixel_around_ML" in the "MultiViewImage.cpp". (for Raytrix R5) Resolution mode = 1: 912x880 (default value) Resolution mode = 2: 832x792 Resolution mode = 3: 744x712 Resolution mode = 4: 656x624 Resolution mode = 5: 568x544 Resolution mode = 6: 480x456

pmode: patch size estimation mode. Please refer to our paper [1][2] for more details. patchEstimationMode, LAPLACIAN_BASE_ALL = 0, (default value) calculating Laplacian sum using all views, LAPLACIAN_BASE_CENTER = 1, calculating Laplacian sum using center view only.

   There are two patch size estimation method using depth information"
   DEPTH_BASE_V1 = 2,
   DEPTH_BASE_V2 = 3,
   Please refer following paper for the details,
   Ref: T. Georgiev and A. Lumsdaine, "Reducing Plenoptic Camera Artifacts", Computer Graphics forum, 2010.

mmode: merge mode for different lens type. Please refer to our paper for more details. mergeMode AVERAGE = 0, merge different lens type using arithmetic mean, GAUSSIAN = 1, merge different lens type using Gaussian weights, BLEND = 2, (default value), merge different lens type using arithmetic mean and Gaussian weights both.

lmode: lens Alignment mode:

   NMF = 0, (N = LensType_1.png, M = LensType_2.png and F = LensType_3.png)
   FMN = 1, (F = LensType_1.png, M = LensType_2.png and N = LensType_3.png) (default)
   NFM = 2, (N = LensType_1.png, F = LensType_2.png and M = LensType_3.png)
   FNM = 3,  (F = LensType_1.png, N = LensType_2.png and M = LensType_3.png)
MFN = 4, (M = LensType_1.png, F = LensType_2.png and N = LensType_3.png) (new)
   MNF = 5 (M = LensType_1.png, N = LensType_2.png and F = LensType_3.png) (new)

   This sets the alignment order of three micro-lens type.
   NMF means {(N)ear, (M)iddle, (F)ar} focal distances for {LensType_1.png, LensType_2.png and LensType_3.png}, respectively.
   For example: NMF represents that the generated lenstype should be arranged with Near focal distance, Middle focal distance, Far distance.

   We recommend you to do a test with single frame input data firstly, while you set the parameters to default value.
   If you find that the generated three lens-type images  (LensType_1.png, LensType_2.png and LensType_3.png)
   are not arranged with default mode (lmode: FMN), please switch it to the suitable one.
   By single frame conversion, you may set the other parameters too.

===========Path Setting=============================================== Calibration_xml: The directory of camera calibration file and the file name.

RawImage_Path: The directory of lenslet raw images and their file name format.

Output_Path: The directory for output multi-view video

===========Pre-processing=============================================== Debayer_mode: Demosaicing method applied to input raw image. 0 means without Demosaicing processing (default) 1 means BG to RGB; 2 means GB to RGB; 3 means RG to RGB; 4 mean GR to RGB. Please refer following link for more details about OpenCV demosaicing precessing. https://docs.opencv.org/3.4.0/de/d25/imgproc_color_conversions.html

Isfiltering: Applying a Gaussian filter on raw images before patch size estimation. 0 means turn off (default), 1 means turn on. Please refer following link for more details about CLAHE: https://docs.opencv.org/3.2.0/d5/daf/tutorial_py_histogram_equalization.html

============Post-processing=============================================== isCLAHE: Applying CLAHE (Contrast Limited Adaptive Histogram Equalization) on rendering multi-view images. 0 means turn off (default), 1 means turn on.

Gamma: This is to adjust parameter of Gamma correction. A positive floating point number is needed, we set this parameter to 1.0 (default). Please refer following link for more details about Gamma correction: https://docs.opencv.org/3.4/d3/dc1/tutorial_basic_linear_transform.html

============Temporal Consistency=============================================== Lambda: This value should be a positive floating point number. It is the coefficient of penalty of temporal consistency. 0 means without penalty of temporal consistency. We recommend setting this parameter to 0.05. (default)

Sigma: This value should be a floating point number. It is the sigma of Gaussian function of temporal consistency. 0 means without Gaussian filter for temporal consistency. (default) If you want to activate this function, we recommend setting this parameter to 0.667.

============Others=============================================== input_model: 0 or 1; 0 means RGB model, 1 means YUV420 model

output_model: 0 or 1; 0 means RGB model, 1 means YUV420 model

start_frame: To set the start frame number of input data.

end_frame: To set the end frame number of input data.