Query regarding Replica Dataset Version & Performance
Closed this issue · 3 comments
Thanks for your great work!
I run the code in dataset of Replica room0, and i get the result of the evaluation metrics. The results are different from the results in paper, but i do not know the reason which the results are different from the results in paper ,which confuses me .
Following is the results i got.
But in the parper, the psnr, ssim ,lpips are 32.86, 0.98, 0.07.
Besides, i get the rendering speed of 103 (in NVIDIA RTX 3090 GPU), but in paper 《Compact 3D Gaussian Splatting For Dense Visual SLAM，it got rendering speed of 125（in NVIDIA RTX 3090Ti GPU）。
So, I have two quesions about the result of the evaluation metrics:
1.why are the results(psnr, ssim, lpips) that i got different from the results in paper?
2.Is the rendering speed is associated with the GPU ？
Looking foward to your reply！
I got this results for room0:
"Final Stats/Avg ATE RMSE": 0.003124260525370642,
"Final Stats/Average PSNR": 32.787879943847656,
"Final Stats/Average Depth RMSE": 0.00498609896749258,
"Final Stats/Average Depth L1": 0.00498609896749258,
"Final Stats/Average MS-SSIM": 0.9768297672271729,
"Final Stats/Average LPIPS": 0.07259822309834701,
Hi, Thanks for trying our code!
One reason for the drastically different results might be the use of a different version of Replica. Please make sure you are using the Replica dataset hosted by the Nice-SLAM authors: https://github.com/spla-tam/SplaTAM?tab=readme-ov-file#replica
There is also a ReplicaV2, which is hosted by VMap, and the images in that dataset are different. Apart from this, I'm unsure why you see such a big difference in performance. We haven't observed this on our end across various machines. Also, are you sure this is the result of Replica Room0?
Re Q2: Yes, rendering speed is associated with GPU. It can also depend on the resolution of images and other particulars like the overhead associated with the wrappings around the renderer.
Yes，you are right. I checked the dataset and found that the dataset of Replica used was different from that in splatam.
I downloaded the dataset whose version is same to splatam ,and i got the results similar to the results shown in the paper.
Thanks for your reply!