/VirtualObjectInsertion

Primary LanguagePython

Virtual Object Insertion

This repository contains the code to generate the object insertion results in paper Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image, CVPR 2020. Please visit our project webpage for more information.

We hope that the code can help reproduce the photorealistic object insertion results in our paper and compare future lighting estimation methods more easily. Please consider citing the paper if you find the code useful. Please contact Zhengqin Li if you have any questions.

Results and Comparisons

The original models were trained by extending the SUNCG dataset with an SVBRDF-mapping. Since SUNCG is not available now due to copyright issues, we are not able to release the original models. Instead, we rebuilt a new high-quality synthetic indoor scene dataset and trained our models on it. We will release the new dataset in the near future. Some object insertion results on real images generated using our network trained on the new dataset are shown below. The quantitative and qualitative comparisons with prior works on object insertion are shown below. Compared with our networks trained on SUNCG-based dataset, our networks trained on new dataset can predict more consistent lighting color and intensity. Therefore, it achieves better performance in user study.

Prerequisites

To run our code, you will need

  • matlab
  • python
  • Renderer: Please find the renderer from this link, which is an Optix-based renderer implemented for this project.

Data

Please download the data from the link. It includes the inverse rendering predictions of the 20 examples from Garon et al. 2019 dataset (Example 1-20) and 4 examples (demo0-3) downloaded from the Internet.

Instructions

To reproduce the 20 object insertion results from Garon et al. 2019 dataset.

  • Download the inverse rendering predictions from the link mentioned above. Unzip the data in the same directory as the code.
  • Open generateXMLread_ref.m. On line 3 and 4, write down the path to your renderer and your python program.
  • Run generateXMLread_ref.m

The program will read the im_*.mat files in each Example* directory to generate the object insertion results.

We also provide the code to generate the object insertion code from scratch. The instructions are as follows:

  • Similarly, download the inverse rendering results from this link
  • Open generateXML.m. On line 4 and 5, enter the path to your renderer and your python program.
  • Several parameters from can be used to adjust the object insertion results
    • meshNewName: The mesh used for object insertion. We provide two meshes: sphere.obj and bunny.obj. You can include your own mesh but you may need to scale and rotate the mesh accordingly so that they will look good.
    • r, g, b: The diffuse color of the inserted object. We set them all to 0.8 in our experiments.
    • roughness: The roughness value of the inserted object. The smaller the roughness value, the more specular the object will be. We set roughness value to 0.2 in all our experiments.
    • scale: The size of the inserted object. The size of the inserted object is decided by the size of the plane you selected from the image (will be explained more detailedly in the following) and the scale parameter.
    • isNewMask: Whether to select new plane or not. Sometimes, after selecting the plane, you may find the rendered object too small or too large. In that case, you can set isNewMask to be false and adjust scale, until you get the satisfactory object insertion result.
  • Run generateXML.m:
    • You will see an image poped out. Choose 4 points on the image as the plane you select. An example is shown below.
    • After choosing the 4 point, the image will be shown again with a red square polygon demonstrating the plane you selected. Choose a point inside the polygon as the place where you would like to inser the object.
    • Done, the rendered result is shown below. The image will be saved as im_new.png in the root directory set on line 3.
  • If you woild like to insert more objects, just change the originName from im.png to im_new.png. Repeat the process and you can insert multiple objects at different locations of a single image.

Limitations

The current code only supports inserting an object on a planar surface. It has not been extended to handle objects in arbitrary 3D geometry yet.