yohanshin/WHAM

Question regarding multi-person videos

tijiang13 opened this issue · 4 comments

tijiang13 commented

Hi Soyong,

Thanks for the great work! I tested the demo and it worked very well. However, when I tried sequences with multiple people, I noticed that the relative positions seemed a bit strange, and all the people appeared to be jammed at the origin in the first frame. Is this the expected behavior?

Best, Tianjian

yohanshin commented

Hi @tijiang13 ,

Each person's global translation is initialized with 0s, so it's expected that each subject starts from the origin. Furthermore, since WHAM predicts each subject individually, the predicted people may not stay in the same world coordinates. One easy (?) solution is,

  1. Take one representative subject among the people.
  2. Construct the camera frame with respect to the reference subject's motion
  3. Re-map other subjects' motion to the reference subject's world frame using camera un-projection.
tijiang13 commented

Thanks @yohanshin!

I also tried that idea last week, however one problem is that (correct me if I am wrong) when I tried to align one subject's motion (in global coord) with the camera, the alignment of the first frame is decent but it drifts away quickly even when calib is provided -- my thought is that when WHAM predicts the translation (in world coordinate) it did not use the calibration explicitly. Do you think that's expected?

Best, Tianjian

yohanshin commented

Yes, since WHAM does not use the SLAM results explicitly, it suffers from drift. I haven't experimented with, but what you can add some post-processing using SLAM results (update the global orientation using the SLAM results and then refine root velocity accordingly). Since SLAM results will be applied equally across subjects, this might be good solution to tie all people into the same coordinate system.

  • 👍1
tijiang13 commented

I was trying that idea too but it appears that the motions (in global coords) become a bit of jittery with further refinement. Perhaps it's just yet another scenario where we need to make trade-off between visual appeal and accuracy. Anyway, thanks @yohanshin for the clarification!