How to generate the Environment Maps on our own dataset?

[rockywind]

Hi, thank you for sharing the great work.
I want to run the TensoIR on our own dataset. The data format of our own data is similar to kitti.
Can you give me some advice?

[Haian-Jin]

Hi,

• For the question about how to run on a new dataset:
  TensoIR only needs images with calibrated poses during training. So you only need to write a dataloader for your own data.
  I recommend you follow the script I provide for the nerf-synthetic dataset and edit the code that this setting uses, which means you should use train_tensoIR_simple.py for training and write your dataloader based on dataLoader/tensoIR_simple.py.
  Please try different values of normals_diff_weight if the result does not look right, as mentioned in the readme.md

• For the question about kitti dataset:
  The project is designed for object-level reconstruction, and I didn't try to run it on a scene-level dataset. So I don't know what will happen if you run it on kitti.

Please feel free to ask if you meet more problems. I am a little busy recently, so my response may delay a little, but I will answer your question when I am available.

[rockywind]

你好，有个问题想请教一下哈。
这是我用自采数据做成neff_synthesic格式训练出来的结果。
这张是nvs_with_brdf的结果。

这张是nvs_with_radiance_field的结果。

这个是训练log

红框是我修改的

1. 想请教一下scale_factor,self.near_far和self.scene_bbox的值应该如何设置？
2. 自采数据场景是230张自车不断向前移动采集的，并不是环绕某个物体拍摄的。
3. 从训练的log看，训练的PSNR是24，测试PSNR是13.8，但是训练集和测试集是相同数据，是不是存在训练和测试不一致呢？

[Haian-Jin]

Hi,

1. If you can't get reasonable radiance field reconstruction, then there must be something wrong with your dataloader. I suggest you to run your data on TensoRF first instead of TensoIR and check if your dataloader can get reasonable novel view synthesis results from the radiance field. Those parameters you just mention are from TensoRF. scale_factor should not be changed. self.near_far and self.scene_bbox can be estimated from the sparse point cloud of your colmap results (if you use colmap to estimate camera poses)

2. According to your pictures, your dataset may be too complex for this project. This work is designed for object-level relighting. The scene where you took pictures is too large. I think you can only get reasonable novel view synthesis results from the radiance field. But it is almost impossible to get good relighting results with this project.

[rockywind]

嗯嗯，十分感谢。
想请教一下是否有适合大场景的relighting的开源代码吗？
我找了Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes发现是没开源的。