Training with own data

[sbharadwajj]

Hi,

I am trying to train NerfingMVS with my own data but I don't understand what is the "depth mask" that is used (folder: raw_data/scene*/depth/). How do I obtain this mask for my own data and is it also essential?

This is not the GT depth map used to train monocular depth network, right? The depth map obtained from MVS of colmap is used as GT, atleast thats what I understood from reading the paper.

[sbharadwajj]

Update: just went through the code and the maps in the depth/ folder are used only for evaluation and nowhere in the training. So we don't require it to train it with our own data, right?

[weiyithu]

Yeah, you're right! The depth GT are only used for evaluation and you do not need them when running your own data.

[sbharadwajj]

But I am still not getting an intuition on why the prediction depth prior from monocular network is less noisier than GT (that we obtain from colmap). I visualized both of them and GT depth map is very noisy compared to the prediction. How is the monocular network able to predict a better depth map?

[weiyithu]

This is the reason why we also output a depth mask from COLMAP instead of using all depths. The depth mask is generated from the fusion step, where geometric check is adopted. Thus, the masked depth is sparse but accurate.

[sbharadwajj]

Oh, I think I have missed this point. So the GT for monocular depth network is not data/scene*/dense/stereo/depth_map/name.geometric.bin?
Where can I find the masked depth?

[weiyithu]

It is in the data/scene*/dense/fused.ply.vis and you must use our version of COLMAP instead of original version.

[sbharadwajj]

Yes, I was able to find more details about the mask in io_utils.py. Thank you so much for patiently answering my questions :)
I just have a few more questions:

1. to synthesize novel views (#5), I took a look at this issue, so do we have to choose the nearest view, get its depth_prior and depth_confidence and then render the novel view?
2. In the results folder that you have shared; there are three subfolders: depth_priors, nerf, filter. Is the final result shown in your paper in Fig 6 from the filter folder? And the folder nerf contains results before the confidence filtering step?

[weiyithu]

Yeah, your understanding is right!