Why define an intrinsicsMLP
Closed this issue · 6 comments
Hi @gengshan-y , thanks for your work.
I want to ask why you define an intrinsicsMLP to learn the focal length and center points while they are already in the configuration file.
https://github.com/lab4d-org/lab4d/blob/24ed2d23d6a53def5baec7f6f43ce4052244502e/lab4d/engine/model.py#L253C16-L253C16
Hi, the intrinsics parameters in the config file are initial guesses based on heuristics defined here.
They could be inaccurate depending on the camera and zooming. As a result, we need to optimize them accordingly. Focal length is likely to change over time due to zoom, so we used a time-dependent MLP for it.
Thanks for your reply. I try to train Lab4d on my own data (a human), the depth loss may collapse and fail to converge while other losses decrease. But it seems it will influence the masks. Do you have any suggestions?
Hi, the default weight of depth loss was set to a small value in lab4d/config.py, which might explain why it did not converge. In fact, depth loss is used most for the sake of background reconstruction. For foreground, I usually disable it as it does not usually contribute much to the optimization.
As long as flow loss and silhouette loss are converging, it should be fine.
Hi, thanks for your explanation. It helps after I set a larger value of depth_wt.
I also tried to train a rigid object (a simple ball), and I found that module.fields.field_params.sdf.bias tends to have a large grad. It will cause the resume from cached weights when checking the global grad norm. It always happens if I set the round=60 and also happens when set the default round. Do you think a rigid object like a ball is too simple so it converges so fast to make this situation happen?
Hi @s0rrymaker77777, the gradient check and weight caching mechanism is to avoid weight explosion (which we found happens more often when skinning deformation fields are used). So one thing you could try is to set the threshold to a larger value, to avoid triggering this mechanism.
Recently there was an improvement on skinning deformation fields (7d72404), and it's likely to improve the stability such that we don't need a low gradient threshold of 5.
Thank you for your thorough explanation!