wanmeihuali/taichi_3d_gaussian_splatting

Is Enable camera pose optimization #83 solved?

TwiceMao opened this issue · 5 comments

Is Enable camera pose optimization #83 solved?
Thx!

The recent finding is that: even for pose optimization only(Given a trained scene), some pose still does not converge. I'm still debugging it.

ttsesm commented

Hi guys,
What do you mean by "enable camera pose optimization"?

Do you mean to solve for the camera pose instead of the image during the inference time or something else?

Thanks.

@ttsesm Yep, as long as the training process is

$$ loss=L(render(gaussian, T_{camera})) $$

$$ gaussian=gaussian-\delta_{lr} \frac{\partial{loss}}{\partial{gaussian}} $$

It shall also be possible to optimize the camera pose:

$$ T_{camera}=T_{camera}-\delta_{lr} \frac{\partial{loss}}{\partial{T_{camera}}} $$

Given a trained scene and a camera pose with noise, we shall be able to optimize the camera pose to remove noise from it. Further, given a list of image-camera pose pairs in which the camera pose is noisy, we might still be able to train the scene.

The following papers may help you learn more about pose optimization in NeRF:
barf: https://arxiv.org/abs/2104.06405
nope-nerf: https://arxiv.org/abs/2212.07388

ptg666 commented

Hi, has the problem of camera pose optimization been solved? @wanmeihuali

Same question here!