Need help about mipnerf ablation for kitti
Closed this issue · 14 comments
Update:
Sorry for the lengthy explanation.
How can we implement 'class-wise' NeRFacto, MipNeRF, and vanilla-NeRF in our code, while ensuring that the latent code works properly for the KITTI scenes?
eg. Do not need the effect where the appearance of the same class is completely uniform.
MipNeRF、NeRF错误信息
I use this method for kitti dataset.
It turns out that:
NeRF: no forground objects (Solved the problem caused by myself, but still undesirable, see another commet below)
_target=NeRFModel,
num_coarse_samples=32,
num_importance_samples=64,
MipNeRF: error
_target=MipNerfModel,
num_coarse_samples=48,
num_importance_samples=96,
How to solve this?
Here is my method:
KITTI_Mars_Depth_class_wise_NVS = MethodSpecification(
config=TrainerConfig(
method_name="mars-kitti-depth-class-wise-nvs",
steps_per_eval_image=STEPS_PER_EVAL_IMAGE,
steps_per_eval_all_images=STEPS_PER_EVAL_ALL_IMAGES,
steps_per_save=STEPS_PER_SAVE,
max_num_iterations=MAX_NUM_ITERATIONS,
save_only_latest_checkpoint=True,
mixed_precision=False,
use_grad_scaler=True,
log_gradients=True,
pipeline=MarsPipelineConfig(
datamanager=MarsDataManagerConfig(
dataparser=MarsKittiDataParserConfig(
use_car_latents=True,
use_depth=True,
car_object_latents_path=Path(
"/root/autodl-tmp/light/mars/kitti/MOT/training/latent/latent_codes06.pt"
),
split_setting="nvs-75",
car_nerf_state_dict_path=Path("/root/autodl-tmp/light/mars/kitti/MOT/training/latent/epoch_670.ckpt"),
scale_factor=0.1,
),
train_num_rays_per_batch=4096,
eval_num_rays_per_batch=4096,
camera_optimizer=CameraOptimizerConfig(mode="off"),
),
model=SceneGraphModelConfig(
background_model=NerfactoModelConfig(),
object_model_template=VanillaModelConfig(
_target=MipNerfModel,
num_coarse_samples=48,
num_importance_samples=96,
# _target=NeRFModel,
# num_coarse_samples=32,
# num_importance_samples=64,
),
object_representation="class-wise", # class-wise
object_ray_sample_strategy="remove-bg",
mono_depth_loss_mult=0.01,
depth_loss_mult=0,
),
),
optimizers={
"background_model": {
"optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
},
"learnable_global": {
"optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
},
"object_model": {
"optimizer": RAdamOptimizerConfig(lr=5e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
},
},
# viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
vis="wandb",
),
description="Neural Scene Graph implementation with vanilla-NeRF model for backgruond and object models.",
)And what I mean is how to use a regular NSG to combine running MipNeRF, NeRF or NeRFacto? Test for ablation.
In urgent need your help!
Looking forward to your reply.
🥺
Here is addition information:
when i use "class-wise" for NerfactoModelConfig
It turns out that all object has the same apperance:
My method:
KITTI_Mars_Depth_class_wise_NVS = MethodSpecification(
config=TrainerConfig(
method_name="mars-kitti-depth-class-wise-nvs",
steps_per_eval_image=STEPS_PER_EVAL_IMAGE,
steps_per_eval_all_images=STEPS_PER_EVAL_ALL_IMAGES,
steps_per_save=STEPS_PER_SAVE,
max_num_iterations=MAX_NUM_ITERATIONS,
save_only_latest_checkpoint=True,
mixed_precision=False,
use_grad_scaler=True,
log_gradients=True,
pipeline=MarsPipelineConfig(
datamanager=MarsDataManagerConfig(
dataparser=MarsKittiDataParserConfig(
use_car_latents=True,
use_depth=True,
car_object_latents_path=Path(
"/root/autodl-tmp/light/mars/kitti/MOT/training/latent/latent_codes06.pt"
),
split_setting="nvs-75",
car_nerf_state_dict_path=Path("/root/autodl-tmp/light/mars/kitti/MOT/training/latent/epoch_670.ckpt"),
scale_factor=0.1,
),
train_num_rays_per_batch=4096,
eval_num_rays_per_batch=4096,
camera_optimizer=CameraOptimizerConfig(mode="off"),
),
model=SceneGraphModelConfig(
background_model=NerfactoModelConfig(),
object_model_template=NerfactoModelConfig(),
object_representation="class-wise", # class-wise
object_ray_sample_strategy="remove-bg",
mono_depth_loss_mult=0.01,
depth_loss_mult=0,
),
),
optimizers={
"background_model": {
"optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
},
"learnable_global": {
"optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
},
"object_model": {
"optimizer": RAdamOptimizerConfig(lr=5e-3, eps=1e-15),
"scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
},
},
# viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
vis="wandb",
),
description="Neural Scene Graph implementation with vanilla-NeRF model for backgruond and object models.",
)I fixed the error about NeRFModel (Caused by myself)
But they are still in same apperance.
pipeline=MarsPipelineConfig(
datamanager=MarsDataManagerConfig(
dataparser=MarsKittiDataParserConfig(
use_car_latents=True,
use_depth=True,
car_object_latents_path=Path(
"/root/light/mars/dataset/kitti/MOT/training/latent/latent_codes06.pt"
),
split_setting="nvs-75",
car_nerf_state_dict_path=Path("/root/light/mars/dataset/kitti/MOT/training/latent/epoch_670.ckpt"),
scale_factor=0.1,
),
train_num_rays_per_batch=4096,
eval_num_rays_per_batch=4096,
camera_optimizer=CameraOptimizerConfig(mode="off"),
),
model=SceneGraphModelConfig(
background_model=NerfactoModelConfig(),
# object_model_template=NerfactoModelConfig(),
object_model_template=VanillaModelConfig(
_target=NeRFModel,
num_coarse_samples=32,
num_importance_samples=64,
# _target=MipNerfModel,
# num_coarse_samples=48,
# num_importance_samples=96,
),
# object_model_template=CarNeRFModelConfig(_target=CarNeRF),
object_representation="class-wise",
object_ray_sample_strategy="remove-bg",
mono_depth_loss_mult=0.01,
depth_loss_mult=0,
),Does it really make sense if you use car_latents with NerfactoModel or VanillaModel?
IIUC, latent codes should only be used with CarNerfModel?
Does it really make sense if you use car_latents with NerfactoModel or VanillaModel? IIUC, latent codes should only be used with CarNerfModel?
@Nplace-su Maybe. Is there any possible way to use "class-wise" just like NSG with MipNeRF or NeRFacto?
@Yurri-hub Why do you want to use "class-wise"
@Nplace-su
In these parts.
It means that "class-wise" will have low model memory occupancy? (In my opinion)
if self.config.object_representation == "class-wise":
object_model_key = [self.get_object_model_name(key) for key in self.object_meta["obj_class"]]
else:
object_model_key = [self.get_object_model_name(key) for key in self.object_meta["scene_obj"]]# TODO(noted by Tianyu LIU): modify various categories of cars
# TODO (wuzr): unifing all configurations
object_models = {
key: self.config.object_model_template.setup(
scene_box=self.scene_box,
num_train_data=self.num_train_data,
object_meta=self.object_meta,
obj_feat_dim=self.config.latent_size if self.use_object_latent_code else 0,
car_latents=self.car_latents,
car_nerf_state_dict_path=self.car_nerf_state_dict_path,
)
for key in object_model_key
}
self.object_models = torch.nn.ModuleDict(object_models)@Yurri-hub Yes, but it seems we can not directly use car_latents with nerfacto/mipnerf model. Maybe you could modify these models like a CarNerf model, however, you still can not use the pretrained latent codes?
Maybe a random latent codes to be optimized is a solution? @Nplace-su
Yes. But i don't think that simple way will give you a good result.
@Nplace-su yeah, that's a problem
Yes. But i don't think that simple way will give you a good result.
@Nplace-su yeah, that's a problem
@Yurri-hub So I'd suggest using the object-wise model if you didn't have very limited memory.
Yes. But i don't think that simple way will give you a good result.
@Nplace-su yeah, that's a problem@Yurri-hub So I'd suggest using the object-wise model if you didn't have very limited memory.
Thanks a lot!
Hi! @Yurri-hub, I am closing this issue for inactivity, but feel free to reopen :)