Evaluation Issues

[DapengFeng]

Thanks for your great work! I still have some issues with the evaluation. Here's a more detailed look:

1. Are the reported PSNR, SSIM, and LIPIS values for all frames or just selected keyframes? The released code only renders and records the keyframes.
2. When calculating rendering metrics, the released code uses undisort_mask to disable invalid regions. What is the distinction between undisort_mask and gt_depth_mask, such as gt_depth > 0? How about using the original image without any mask for evaluation?

I really appreciate hearing from you.

[liquorleaf]

1. When evaluating, we use another self-wirtten Python tool to render the reconstructed point cloud for all frames, not only Keyframes. However the tool is too naive and even ugly in programming so we didn't release it. You can ask @HuajianUP for more information about the code of our evaluation tool.
2. We undistort all gt images simply because our Gaussian Splatting backend cannot deal with distortion. It can only render undistorted views. So we use undistort all images for photorealistic evaluation. This is also a limitation of the original 3D Gaussian Splatting.
3. As for gt depth, we use thresholds to filter out extremely small or large depth just for accuracy consideration. This is common for industrial depth cameras such as the Intel Realsense D455.

[DapengFeng]

I really appreciate your help. I've been studying different GS-SLAM repositories and noticed the various evaluation techniques used. Your method stands out with better qualitative results compared to others, even under similar PSNR, SSIM, and LPIPS reported in the papers. So I would like to request the evaluation tool for fair comparison with others at a quantitative level. Thank you for your assistance! @HuajianUP

[HuajianUP]

Hi, Dapeng. I have just made the evaluation code publicly accessible. Please refer to the updated README.md file for detailed information. I hope this code will be beneficial to you during the evaluation process.

However, I would like to note that the evaluation code has not been cleaned up and well-tested, especially for the Python environment. You are welcome to contribute.