The evaluation accuracy problem on Hypersim

[tsuJohc]

Hi, thanks for releasing such great work! I am testing the provided model on the Hypersim.

However, I found the accuracy is not very good. Therefore, I want to have a verification with you: if the bad results are reasonable or there is something wrong in my processing.

I test on the ai_001_010/cam_00 of the Hypersim, and use the frame.xxxx.normal_cam.png as ground truth.
The accuracy is

total_iter (# 78198048): ai_001_010/cam_00:
mean median rmse 5 7.5 11.25 22.5 30
52.066 41.052 69.400 8.312 14.636 26.022 40.487 43.882

Besides, I don't know which camera coordinate system (opencv/opengl) you use and camera in the Hypersim is the opengl coordinate system ( x-axis points right, the positive y-axis points up, and the positive z-axis points away from where the camera is looking.) Therefore, I convert the GT normal to the opencv-camera coordinate system, where I negate the y-axis and z-axis , and evaluate again. The accuracy becomes worse:

total_iter (# 78198048): ai_001_010/cam_00
mean median rmse 5 7.5 11.25 22.5 30
170.977 177.555 171.943 0.000 0.000 0.000 0.001 0.002

Here is a example contains the ground truth, input color image and the results of DSINE.

P.S. I have provide DSINE with the intrinsic of Hypersim.

[baegwangbin]

Hi, sorry for the delayed response.

The coordinate system is: X-right / Y-down / Z-front.

The surface normals vectors are defined as the outward normals.
For example, for a flat wall facing the camera, the normal would be (0, 0, +1).
Invert the vector to obtain the inward normals.

Hypersim uses a different coordinate system so you need to adjust it before evaluation.

[tsuJohc]

Thanks, It helps a lot!

[BlingHe]

Thanks, It helps a lot!

I also encountered the same problem. How did you solve it?