potential bug: Backward of depth loss
leo-frank opened this issue · 7 comments
I wonder why use projmatrix to compute gradients ?
In fact, the cam.z is computed from p_view.z, see depth forward.
Can you verify this potential bug ?
p_view has no relation with projmatrix.
p_view is actually transformed from p_orig by viewmatrix, see p_view calculation
Hello, thank you for the valuable comment.
Actually, we slightly modify the rasterization code from the original repo named diff-gaussian-rasterization.
The lines that you have pointed out are same as the lines in the original repo.
I think it would be better to open the issue or make an pull request to that repository.
By the way I will also check if there is a redundant part in that code, too.
Sorry, I didn't express my question clearly. Please let me retry again.
Firstly, I know that your rasterization codes are based on original repo, and i think you are actually adding depth forward functionality and backward functionality.
Secondly, I think there is a bug in your depth backward implementation. In here , we want to compute dL_dmean3D from dL_camz.
The way that you do is utilizing information from projmatrix which operates on points_xyz[global_id], see m_hom .
But actually, the cam_z(also known as p_view.z) is computed from only viewmatrix. p_view calculation in forward.cu.
So, when we compute dL_dmean3D from dL_camz, there should be no stuff about projmatrix., and should only has relation with viewmatrix.
Can you verify this again? Thanks a lot !
@leo-frank
You're right. Here's the story:
We created the 'backward' function for use in another project of ours. Amid simultaneous development, we accidentally brought in and used an unchanged previous version that hadn't fixed errors. We didn't bother fixing it in the LucidDreamer project since 'backward' wasn't being used there.
But now that I've heard your point, we recognize it could lead to misunderstandings. I'll make the necessary edits by either removing or fixing the 'backward' as you suggested. Thanks for the comment.
@robot0321 Hi, really thanks for your reply!
I wonder if there is a repository that contains correct depth backward implementation, like the one that you say "another project of ours" ?
@leo-frank
Unfortunately, we are under-reviewed, so we will distribute the repo later.
If you can not wait more, please refer other projects such as
- SparseGS: Real-Time 360° Sparse View Synthesis using Gaussian Splatting
- FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting
- pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction
- Splatter Image: Ultra-Fast Single-View 3D Reconstruction
Or see here for easy links!
(Note: I did not check the codes in detail)
Really thanks for your reply!
I just implement a repo that contains correct depth backward implementation: https://github.com/leo-frank/diff-gaussian-rasterization-depth. Welcome to try it!