FieldHeadNames.SEMANTICS KeyError when training Mars Recon on KITTI-Scene05

[amoghskanda]

Thanks for this great work. I was training from scratch on Scene05 of KITTI using depth_maps and panoptic_maps generated using mini-omnidata and panoptic generator respectively, for scene05, and placed in the respective directories. I have set use_semantic to True in mars_kitti_dataparser.py and have provided the correct path as well. When I try training, scene_graph.py line498 throws a KeyError( KeyError: <FieldHeadNames.SEMANTICS: 'semantics'>). I tried debugging using pdb, output_background['field_outputs'] has no key by the name FieldHeadNames.SEMANTICS. Are there any changes to be made to Nerfacto to incorporate the semantic maps? I keep getting this error as well AttributeError: 'NerfactoModel' object has no attribute 'num_semantics' but I fixed it by setting a value to num_semantics . I'm guessing there's a semantics error wrt NerfactoModel

[JiantengChen]

Hi! @amoghskanda

Thanks for your reply. It seems like you are using NeRFactoModelConfig() for the background model. Maybe you can refer to the config below.

KITTI_Recon_Mars_Car_Depth = MethodSpecification(
    config=TrainerConfig(
        method_name="mars-kitti-car-depth-recon",
        steps_per_eval_image=STEPS_PER_EVAL_IMAGE,
        steps_per_eval_all_images=STEPS_PER_EVAL_ALL_IMAGES,
        steps_per_save=STEPS_PER_SAVE,
        max_num_iterations=MAX_NUM_ITERATIONS,
        save_only_latest_checkpoint=False,
        mixed_precision=False,
        use_grad_scaler=True,
        log_gradients=True,
        pipeline=MarsPipelineConfig(
            datamanager=MarsDataManagerConfig(
                dataparser=MarsKittiDataParserConfig(
                    use_car_latents=True,
                    use_depth=True,
                    use_semantic=True,
                    car_object_latents_path=Path(
                        "/DATA_EDS/liuty/ckpts/pretrain/car_nerf/kitti_mot/latents/latent_codes05.pt"
                    ),
                    semantic_mask_classes=["Van", "Undefined"],
                    semantic_path=Path("/DATA_EDS/wuzr/data/kitti-MOT/panoptic_maps"),
                    split_setting="reconstruction",
                    car_nerf_state_dict_path=Path(
                        "/DATA_EDS/liuty/ckpts/pretrain/car_nerf/epoch_670.ckpt"
                    ),
                    scale_factor=0.05,
                ),
                train_num_rays_per_batch=4096,
                eval_num_rays_per_batch=4096,
                camera_optimizer=CameraOptimizerConfig(mode="off"),
            ),
            model=SceneGraphModelConfig(
                background_model=SemanticNerfWModelConfig(
                    num_proposal_iterations=1,
                    num_proposal_samples_per_ray=[48],
                    num_nerf_samples_per_ray=97,
                    use_single_jitter=False,
                    semantic_loss_weight=0.1,
                ),
                object_model_template=CarNeRFModelConfig(_target=CarNeRF),
                object_representation="class-wise",
                object_ray_sample_strategy="remove-bg",
                mono_depth_loss_mult=0.01,
                depth_loss_mult=0,
            ),
        ),
        optimizers={
            "background_model": {
                "optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
                "scheduler": ExponentialDecaySchedulerConfig(
                    lr_final=1e-5, max_steps=200000
                ),
            },
            "learnable_global": {
                "optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
                "scheduler": ExponentialDecaySchedulerConfig(
                    lr_final=1e-5, max_steps=200000
                ),
            },
            "object_model": {
                "optimizer": RAdamOptimizerConfig(lr=5e-3, eps=1e-15),
                "scheduler": ExponentialDecaySchedulerConfig(
                    lr_final=1e-5, max_steps=200000
                ),
            },
        },
        # viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
        vis="wandb",
    ),
    description="Neural Scene Graph implementation with vanilla-NeRF model for backgruond and object models.",
)

[amoghskanda]

Hey!Thanks, I made the change from Nerfacto to SemanticNerfWModel and now I'm able to train it on a dataset with panoptic maps.

[amoghskanda]

@xBeho1der my depth maps are 3-channel(rgb) and not just black and white. Should I modify something before training or will that be fine?

[wuzirui]

answered in #152