How to train DyNeRF scenes ?

Hi, thanks for providing the code ! I am trying to train DyNeRF scenes but I keep on stucking at the dataloader part.

 'isg': False,
 'isg_step': -1,
 'linear_decoder': False,
 'logdir': './logs/realdynamic',
 'ndc': True,
 'num_samples': 48,
 'num_steps': 90001,
 'optim_type': 'adam',
 'plane_tv_weight': 0.0002,
 'plane_tv_weight_proposal_net': 0.0002,
 'proposal_net_args_list': [{'num_input_coords': 4,
                             'num_output_coords': 8,
                             'resolution': [128, 128, 128, 150]},
                            {'num_input_coords': 4,
                             'num_output_coords': 8,
                             'resolution': [256, 256, 256, 150]}],
 'save_every': 30000,
 'save_outputs': True,
 'scene_bbox': [[-3.0, -1.8, -1.2], [3.0, 1.8, 1.2]],
 'scheduler_type': 'warmup_cosine',
 'single_jitter': False,
 'time_smoothness_weight': 0.001,
 'time_smoothness_weight_proposal_net': 1e-05,
 'train_fp16': True,
 'use_proposal_weight_anneal': True,
 'use_same_proposal_network': False,
 'valid_every': 30000}
2023-01-25 15:42:09,846|    INFO| Loading Video360Dataset with downsample=2.0
Loading train data:  53%|███████████████████████████████████████████████████████████████████████████▎                                                                   | 10/19 [00:43<00:34,  3.84s/it

I saw the recommendation to train these scenes with data_downsample=4 for one step. How exactly can I do that ? Should I manually change 'num_steps': 1 ?

Thanks for trying the code out so quickly! To set up the importance sampling, you would set data_downsample to 4 and num_steps to 1 in the config you're using. Then once it finishes you can reset to the original parameters and run again to actually train. Be aware that the DyNeRF dataset is much larger than the other datasets, and it can take several minutes to load (the other datasets load in a few seconds).

How does K-Planes remember the importance sampling across the two runs? Does it save any file containing the importance sampling weights during the first run (with downsample=4) and load these weights during the second run (with downsample=2)?

Yes, it saves files called isg_weights.pt and ist_weights.pt with the importance sampling weights. I included these files for some of the scenes in the raw outputs folder on Google Drive, in case that is useful.

Great! Thank you, for the clarification. One follow-up question. I am running out of memory (CPU RAM) when creating the isg_weights.pt and ist_weights.pt files for cutbeef scene with downsample=4. Would the performance take a serious hit if I create the importance sampling weights with downsample=8? Would you suggest any alternative to overcome the memory issue?

You could try with downsample 8; I don't expect it would hurt performance that much but there's a risk I might have hardcoded the 4 somewhere, so it might error but hopefully not. It's also not that much worse if you train without any importance sampling at all, though you will lose some details in the dynamic regions.

I generated weights with down=8 and trained the model with down=2. The code didn't throw any error. So, I guess you haven't hard coded 4!

I'm going through something similar and saw the last comment. Just wanted to add that this bit is hardcoded

K-Planes/plenoxels/datasets/video_datasets.py

Line 153 in 7e3a82d

weights_subsampled = int(4 / downsample)