Projecting 3D keypoints onto original image

[wclincke]

I am encountering a challenge with visualizing projected 3D keypoints onto a 2D image. The keypoints appear to be incorrectly rotated and scaled, leading to a misalignment with the corresponding subjects in the image.

I am adding this functionality to the demo.py file. Here's the code snippet that performs the perspective projection of the 3D keypoints:

# Compute the 3D keypoints tensor
keypoints_3d = out['pred_keypoints_3d'].cpu().numpy()
torch.tensor(keypoints_3d).to('cuda')
     
# Perform perspective projection
projected_2d_keypoints = perspective_projection(
    keypoints_3d_tensor, 
    translation=pred_cam_t, 
    focal_length=focal_length, 
    camera_center=camera_center
)

The expected behavior is that the perspective_projection function from here would handle the correct rotation and scaling. However, the keypoints do not align with the subjects as expected.

Here is an image that shows the current output of the projection. The green lines represent the keypoints projected onto the 2D image, which clearly do not align correctly.

Could anyone guide me on how to properly adjust the rotation and scaling? Insights on how to correct this within the context of the demo.py file would be extremely helpful.

Thanks in advance for your support!

[geopavlakos]

You need to use the pred_cam_t_full variable instead of pred_cam_t for the projection on the full image to work properly.