Questions about calculating projection matrices

Hi, I noticed that your CUDA code uses full_proj_transform

gaussian_surfels/gaussian_renderer/__init__.py

Line 47 in c1cbb40

projmatrix=viewpoint_camera.full_proj_transform,

, which is derived from projection_matrix

Line 97 in c1cbb40

    
           self.full_proj_transform = (self.world_view_transform.unsqueeze(0).bmm(self.projection_matrix.unsqueeze(0))).squeeze(0)

but you does not take into account the principal point when calculating projection_matrix

gaussian_surfels/scene/cameras.py

Line 72 in c1cbb40

    
           self.projection_matrix = getProjectionMatrix(self.znear, self.zfar, FoVx, FoVy, prcppoint).transpose(0,1).to(data_device)

gaussian_surfels/utils/graphics_utils.py

Line 66 in c1cbb40

def getProjectionMatrix(znear, zfar, fovX, fovY, prcp):

So even though you used the principal point in the CUDA code, using the full_project_matrix may still result in imprecise results

gaussian_surfels/submodules/diff-gaussian-rasterization/cuda_rasterizer/forward.cu

Line 254 in c1cbb40

float4 p_hom = transformPoint4x4(p_orig, projmatrix);

Is what I am considering correct?

Hi, yes the projection matrix is imprecise as we consider the principal point here, but it seems better to do it in projection matrix.

Hi, I also noticed that you use principal point to recover point_image, my point is that the input p_proj is calculated from full_proj_transform, and full_proj_transform is not suitable for the imperfect pinhole camera, so the recovered point_image is still not accurate.

Hi, I think you are right. I didn't think deep here. I refered to this and updated the projection matrix. The previous modification in CUDA is now reversed. Thank you very much for your remider! I am not very sure if this is properly fixed now, please feel free to get in touch.

That's a good idea!