train custom dataset

[q-eldar]

Hi!
Thanks for your amazing work. So I want to train it on custom dataset. Is it possible to generate dataset without segments masks, train and get segmentation masks? If masks are necessary how many of them are required ? I have 144 images of dataset + reconstructed poses with colmap.

[zhiwenfan]

Hi,

masks, segments are not used for training nerf-sos. They are used for semantic-nerf. You can first use colmap to do sparse reconstruction, use LLFF to get the necessary meta data, then run gen_dataset.py with removing the requirements of "masks" and "segments".
I will provide a detailed tutorial for custom data recently.

[q-eldar]

Detailed tutorial would be really helpful!

Meanwhile can you please explain, should I remove masks from gen_dataset.py and load_llff.py (currently Im trying generate masks for llff dataset format) including these lines for training only:

    np.save(os.path.join(output_path, 'rays_train.npy'), rays_train)
    np.save(os.path.join(output_path, 'rgbs_train.npy'), rgbs_train)
    np.save(os.path.join(output_path, 'masks_train.npy'), masks_train) <-
    np.save(os.path.join(output_path, 'rays_val.npy'), rays_val)
    np.save(os.path.join(output_path, 'rgbs_val.npy'), rgbs_val)
    np.save(os.path.join(output_path, 'masks_val.npy'), masks_val) <-
    np.save(os.path.join(output_path, 'rays_test.npy'), rays_test)
    np.save(os.path.join(output_path, 'rgbs_test.npy'), rgbs_test)
    np.save(os.path.join(output_path, 'masks_test.npy'), masks_test) <-

[twahakabika]

Hello

Congratulations on your good work. Did you manage to make a video on how to generate custom data?