How to improve estimation of arm and hand angles in extreme scenarios?
Closed this issue · 4 comments
As I already checked many repos and papers, that work with SMPL fitting, I didn't get good apropriate result in hand/arm/forearm tracking in negative extreme angles. Here are few screenshots (left -- 4d humans, middle and right WHAM w/o and w postprocessing). Is there any way how to improve this or finetune the model?
Overall 4d humans have generally best results IMHO, except these issues with extreme angles during f.e. tennis serve:
If you want to potentially refine the output from 4D Humans, you could use our slahmr work, which adds some overhead in the runtime but it typically leads to more precise results.
Could you also share the original RGB video you are using for this inference?
@geopavlakos thank you! Btw I just found slahmr 30 mins ago. So will try and let you know how it'll work!
Here is the RGB video: https://drive.google.com/file/d/11I6hfR0BkOGnqS1wc0hnQAngtTZsQTEM/view?usp=sharing
@geopavlakos I also would say not to refine result itself, but to refine model with additional dataset, that can be captured and transformed to SMPL type.
In terms of finetuning the model, that is definitely possible, if you have this type of data (i.e., images and corresponding 3D ground truth in SMPL format). We have not tested finetuning HMR2.0 with data from a specific activity, but I expect it would help reconstructing poses from that particular activity more accurately.