reconstruction result with pretrained model looks strange
Closed this issue · 9 comments
Hi, below are the render results with pretrained model (outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/nerfstudio_models/step-000400000.ckpt), they are different with the results you posted.
There are shadows on the road in the background, and background pixels in object_rgb (red box).
Could you give some instructions?
Besides, I'm training the scene (kitti-0006) from scrach, the rgb results seems to have the same problem, and the depth prediction is also abnormal.
https://wandb.ai/hemp110/nerfstudio-project/runs/iydafnkm?workspace=user-hemp110
Hi! @hemp110
- The posted visualization result was not rendered with the checkpoint on the master branch now. Maybe training a longer sequence will help. Below is the background rgb from a longer sequence.
- For depth prediction, #92 might help. For the monocular depth supervision visualization, ground truth depth and predicted truth are not of identical scale, clipping operation probably should not be applied.
Hi, below are the render results with pretrained model (outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/nerfstudio_models/step-000400000.ckpt), they are different with the results you posted. There are shadows on the road in the background, and background pixels in object_rgb (red box). Could you give some instructions?
@xBeho1der thanks for your replay, but the result in first post is rendered with checkpoint on the master branch. I downloaded it just 3 or 4 days ago, and render with command
python scripts/cicai_render.py --load-config outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/config.yml --rendered_output_names rgb depth background background_depth objects_rgb objects_depth --output-format images --output-path outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/rendered_images
In config.yml
I only modified data
, car_nerf_state_dict_path
and car_object_latents_path
to my path.
When rendering, I met errors in loading checkpoint
RuntimeError: Error(s) in loading state_dict for SceneGraphModel:
Missing key(s) in state_dict: "background_model.lpips.net.scaling_layer.shift", "background_model.lpips.net.scaling_layer.scale", "object_models.object_class_0.lpips.net.scaling_layer.shift", "object_models.object_class_0.lpips.net.scaling_layer.scale", "object_models.object_class_2.lpips.net.scaling_layer.shift", "object_models.object_class_2.lpips.net.scaling_layer.scale", "lpips.net.scaling_layer.shift", "lpips.net.scaling_layer.scale".
But I didn't find any similar issues in this repo. After searching in web, i fixed it by modifying lpips.py
Is my problem related with my modification?
I tried another pretrained model in the repo (kitti-nvs-75) by running
python scripts/cicai_render.py --load-config outputs/kitti-nvs-75/mars-kitti-car-depth-nvs/2023-10-31_111111/config.yml --rendered_output_names rgb depth background background_depth objects_rgb objects_depth --output-format images --output-path outputs/kitti-nvs-75/mars-kitti-car-depth-nvs/2023-10-31_111111/rendered_images
the rendered results have the same problem. Maybe the shadow in background of kitti-nvs-75
is lighter than kitti-recon-65-120
?
Hi, below are the render results with pretrained model (outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/nerfstudio_models/step-000400000.ckpt), they are different with the results you posted. There are shadows on the road in the background, and background pixels in object_rgb (red box). Could you give some instructions?
@xBeho1der thanks for your replay, but the result in first post is rendered with checkpoint on the master branch. I downloaded it just 3 or 4 days ago, and render with command
python scripts/cicai_render.py --load-config outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/config.yml --rendered_output_names rgb depth background background_depth objects_rgb objects_depth --output-format images --output-path outputs/kitti-recon-65-120/mars-kitti-car-depth-recon/2023-10-30_212654/rendered_images
In
config.yml
I only modifieddata
,car_nerf_state_dict_path
andcar_object_latents_path
to my path. When rendering, I met errors in loading checkpointRuntimeError: Error(s) in loading state_dict for SceneGraphModel: Missing key(s) in state_dict: "background_model.lpips.net.scaling_layer.shift", "background_model.lpips.net.scaling_layer.scale", "object_models.object_class_0.lpips.net.scaling_layer.shift", "object_models.object_class_0.lpips.net.scaling_layer.scale", "object_models.object_class_2.lpips.net.scaling_layer.shift", "object_models.object_class_2.lpips.net.scaling_layer.scale", "lpips.net.scaling_layer.shift", "lpips.net.scaling_layer.scale".
But I didn't find any similar issues in this repo. After searching in web, i fixed it by modifying
lpips.py
Is my problem related with my modification?
Hi! @hemp110
Your operation and modification seem to be fine. I use the same checkpoint and command to render but I haven't encountered this problem.
hi @xBeho1der
I suddenly remember that my server is with CUDA 11.4 instead of 11.7. I forgot it since I can install and run mars and nerfstudio without error. Do you think this is the main reason?
I am trying to update CUDA to 11.7, but it's a hard work for me because I'm using cloud computing resources.
Hi @hemp110, the difference in the runtime environment (including CUDA version, torch version ...) may cause changes in performance. I would suggest training from scratch with the same setting and see if the results are different. You could also refer to the wandb reports we provided.
Besides, I'm training the scene (kitti-0006) from scrach, the rgb results seems to have the same problem, and the depth prediction is also abnormal. https://wandb.ai/hemp110/nerfstudio-project/runs/iydafnkm?workspace=user-hemp110
@wuzirui as in this post, I trained from scratch with the same setting (CUDA 11.4). The key figures, such as train/eval loss, eval psnr/ssim/lpips are basically the same as the wandb you provided. However, the rendered images suffer from the shadow problem.
This is the wandb you provided
This is my wandb
hi @xBeho1der I suddenly remember that my server is with CUDA 11.4 instead of 11.7. I forgot it since I can install and run mars and nerfstudio without error. Do you think this is the main reason? I am trying to update CUDA to 11.7, but it's a hard work for me because I'm using cloud computing resources.
Hi @hemp110,
in case you are having doubts on the cuda version that might lead to the shadows on the streets in the background images, I'm also having the same shadows while I'm using cuda 11.6. In my opinion, the cuda version doesn't make any difference. Maybe train the model from a longer sequence would help(but I haven't tried).