reconstruction result with pretrained model looks strange

[hemp110]

Hi, below are the render results with pretrained model (outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/nerfstudio_models/step-000400000.ckpt), they are different with the results you posted.
There are shadows on the road in the background, and background pixels in object_rgb (red box).
Could you give some instructions?

[hemp110]

Besides, I'm training the scene (kitti-0006) from scrach, the rgb results seems to have the same problem, and the depth prediction is also abnormal.
https://wandb.ai/hemp110/nerfstudio-project/runs/iydafnkm?workspace=user-hemp110

[JiantengChen]

Hi! @hemp110

1. The posted visualization result was not rendered with the checkpoint on the master branch now. Maybe training a longer sequence will help. Below is the background rgb from a longer sequence.

2. For depth prediction, #92 might help. For the monocular depth supervision visualization, ground truth depth and predicted truth are not of identical scale, clipping operation probably should not be applied.

[hemp110]

Hi, below are the render results with pretrained model (outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/nerfstudio_models/step-000400000.ckpt), they are different with the results you posted. There are shadows on the road in the background, and background pixels in object_rgb (red box). Could you give some instructions?

@xBeho1der thanks for your replay, but the result in first post is rendered with checkpoint on the master branch. I downloaded it just 3 or 4 days ago, and render with command

python scripts/cicai_render.py --load-config outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/config.yml --rendered_output_names rgb depth background background_depth objects_rgb objects_depth --output-format images --output-path outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/rendered_images

In config.yml I only modified data, car_nerf_state_dict_path and car_object_latents_path to my path.
When rendering, I met errors in loading checkpoint

RuntimeError: Error(s) in loading state_dict for SceneGraphModel:
        Missing key(s) in state_dict: "background_model.lpips.net.scaling_layer.shift", "background_model.lpips.net.scaling_layer.scale", "object_models.object_class_0.lpips.net.scaling_layer.shift", "object_models.object_class_0.lpips.net.scaling_layer.scale", "object_models.object_class_2.lpips.net.scaling_layer.shift", "object_models.object_class_2.lpips.net.scaling_layer.scale", "lpips.net.scaling_layer.shift", "lpips.net.scaling_layer.scale".

But I didn't find any similar issues in this repo. After searching in web, i fixed it by modifying lpips.py

Is my problem related with my modification?

[hemp110]

I tried another pretrained model in the repo (kitti-nvs-75) by running

python scripts/cicai_render.py --load-config outputs/kitti-nvs-75/mars-kitti-car-depth-nvs/2023-10-31_111111/config.yml --rendered_output_names rgb depth background background_depth objects_rgb objects_depth --output-format images --output-path outputs/kitti-nvs-75/mars-kitti-car-depth-nvs/2023-10-31_111111/rendered_images

the rendered results have the same problem. Maybe the shadow in background of kitti-nvs-75 is lighter than kitti-recon-65-120?

[JiantengChen]

Hi, below are the render results with pretrained model (outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/nerfstudio_models/step-000400000.ckpt), they are different with the results you posted. There are shadows on the road in the background, and background pixels in object_rgb (red box). Could you give some instructions?

@xBeho1der thanks for your replay, but the result in first post is rendered with checkpoint on the master branch. I downloaded it just 3 or 4 days ago, and render with command

python scripts/cicai_render.py --load-config outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/config.yml --rendered_output_names rgb depth background background_depth objects_rgb objects_depth --output-format images --output-path outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/rendered_images

In config.yml I only modified data, car_nerf_state_dict_path and car_object_latents_path to my path. When rendering, I met errors in loading checkpoint

RuntimeError: Error(s) in loading state_dict for SceneGraphModel:
        Missing key(s) in state_dict: "background_model.lpips.net.scaling_layer.shift", "background_model.lpips.net.scaling_layer.scale", "object_models.object_class_0.lpips.net.scaling_layer.shift", "object_models.object_class_0.lpips.net.scaling_layer.scale", "object_models.object_class_2.lpips.net.scaling_layer.shift", "object_models.object_class_2.lpips.net.scaling_layer.scale", "lpips.net.scaling_layer.shift", "lpips.net.scaling_layer.scale".

But I didn't find any similar issues in this repo. After searching in web, i fixed it by modifying lpips.py Is my problem related with my modification?

Hi! @hemp110

Your operation and modification seem to be fine. I use the same checkpoint and command to render but I haven't encountered this problem.

[hemp110]

hi @xBeho1der
I suddenly remember that my server is with CUDA 11.4 instead of 11.7. I forgot it since I can install and run mars and nerfstudio without error. Do you think this is the main reason?
I am trying to update CUDA to 11.7, but it's a hard work for me because I'm using cloud computing resources.

[wuzirui]

Hi @hemp110, the difference in the runtime environment (including CUDA version, torch version ...) may cause changes in performance. I would suggest training from scratch with the same setting and see if the results are different. You could also refer to the wandb reports we provided.

[hemp110]

Besides, I'm training the scene (kitti-0006) from scrach, the rgb results seems to have the same problem, and the depth prediction is also abnormal. https://wandb.ai/hemp110/nerfstudio-project/runs/iydafnkm?workspace=user-hemp110

@wuzirui as in this post, I trained from scratch with the same setting (CUDA 11.4). The key figures, such as train/eval loss, eval psnr/ssim/lpips are basically the same as the wandb you provided. However, the rendered images suffer from the shadow problem.
This is the wandb you provided


This is my wandb

[mengZhang9611]

hi @xBeho1der I suddenly remember that my server is with CUDA 11.4 instead of 11.7. I forgot it since I can install and run mars and nerfstudio without error. Do you think this is the main reason? I am trying to update CUDA to 11.7, but it's a hard work for me because I'm using cloud computing resources.

Hi @hemp110,

in case you are having doubts on the cuda version that might lead to the shadows on the streets in the background images, I'm also having the same shadows while I'm using cuda 11.6. In my opinion, the cuda version doesn't make any difference. Maybe train the model from a longer sequence would help(but I haven't tried).