FieldHeadNames.SEMANTICS KeyError when training Mars Recon on KITTI-Scene05
Closed this issue · 4 comments
Thanks for this great work. I was training from scratch on Scene05 of KITTI using depth_maps and panoptic_maps generated using mini-omnidata and panoptic generator respectively, for scene05, and placed in the respective directories. I have set use_semantic to True in mars_kitti_dataparser.py
and have provided the correct path as well. When I try training, scene_graph.py
line498 throws a KeyError( KeyError: <FieldHeadNames.SEMANTICS: 'semantics'>). I tried debugging using pdb, output_background['field_outputs'] has no key by the name FieldHeadNames.SEMANTICS. Are there any changes to be made to Nerfacto to incorporate the semantic maps? I keep getting this error as well AttributeError: 'NerfactoModel' object has no attribute 'num_semantics' but I fixed it by setting a value to num_semantics . I'm guessing there's a semantics error wrt NerfactoModel
Hi! @amoghskanda
Thanks for your reply. It seems like you are using NeRFactoModelConfig()
for the background model. Maybe you can refer to the config below.
KITTI_Recon_Mars_Car_Depth = MethodSpecification(
config=TrainerConfig(
method_name="mars-kitti-car-depth-recon",
steps_per_eval_image=STEPS_PER_EVAL_IMAGE,
steps_per_eval_all_images=STEPS_PER_EVAL_ALL_IMAGES,
steps_per_save=STEPS_PER_SAVE,
max_num_iterations=MAX_NUM_ITERATIONS,
save_only_latest_checkpoint=False,
mixed_precision=False,
use_grad_scaler=True,
log_gradients=True,
pipeline=MarsPipelineConfig(
datamanager=MarsDataManagerConfig(
dataparser=MarsKittiDataParserConfig(
use_car_latents=True,
use_depth=True,
use_semantic=True,
car_object_latents_path=Path(
"/DATA_EDS/liuty/ckpts/pretrain/car_nerf/kitti_mot/latents/latent_codes05.pt"
),
semantic_mask_classes=["Van", "Undefined"],
semantic_path=Path("/DATA_EDS/wuzr/data/kitti-MOT/panoptic_maps"),
split_setting="reconstruction",
car_nerf_state_dict_path=Path(
"/DATA_EDS/liuty/ckpts/pretrain/car_nerf/epoch_670.ckpt"
),
scale_factor=0.05,
),
train_num_rays_per_batch=4096,
eval_num_rays_per_batch=4096,
camera_optimizer=CameraOptimizerConfig(mode="off"),
),
model=SceneGraphModelConfig(
background_model=SemanticNerfWModelConfig(
num_proposal_iterations=1,
num_proposal_samples_per_ray=[48],
num_nerf_samples_per_ray=97,
use_single_jitter=False,
semantic_loss_weight=0.1,
),
object_model_template=CarNeRFModelConfig(_target=CarNeRF),
object_representation="class-wise",
object_ray_sample_strategy="remove-bg",
mono_depth_loss_mult=0.01,
depth_loss_mult=0,
),
),
optimizers={
"background_model": {
"optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(
lr_final=1e-5, max_steps=200000
),
},
"learnable_global": {
"optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(
lr_final=1e-5, max_steps=200000
),
},
"object_model": {
"optimizer": RAdamOptimizerConfig(lr=5e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(
lr_final=1e-5, max_steps=200000
),
},
},
# viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
vis="wandb",
),
description="Neural Scene Graph implementation with vanilla-NeRF model for backgruond and object models.",
)
Hey!Thanks, I made the change from Nerfacto to SemanticNerfWModel and now I'm able to train it on a dataset with panoptic maps.
@xBeho1der my depth maps are 3-channel(rgb) and not just black and white. Should I modify something before training or will that be fine?