Predicted 3D poses

[a-kungwani]

Hi @SoroushMehraban I used your demo code to predict some 3D poses on my inference video. After prediction, I couldn't interpret the pose results. I read some previous issues regarding this and found that predictions are in camera coordinate system. I have attached a screenshot of predicted pose for a particular frame. The pose in the screenshot are before applying the rotation function. The demo code makes the pelvic joint(0th) pose as zero and predict the rest all joint pose in reference to the pelvic joint?

I request you to help me with some clarification for my doubts.

[SoroushMehraban]

Hi @a-kungwani,
Some previous works such as STCFormer gives you the absolution positions in meters when you use them. But the issue with those models is that during pretraining the model only sees sequences captured with 4 cameras in a laboratory environment and the subject is always within a range of distances from the camera. Meaning that at inference time when you give them a new subject (potentially with a different height) captured with a new camera with a different distance and camera viewpoint, the models are probably give you some measurements that are not true.

So we have an ill-posed problem, a person tall away from the camera has the same 2D pose sequence as the person short close to the camera. To mitigate the issue, we follow MotionBERT and our output:

• Always is centered around bottom torso.
• The output is normalized in [-1, 1]. And even after denormalization it doesn't give you a measure in meters (because it's impossible to get it from a single camera). So instead it produces a unit that is scale invariant, and both tall and short persons are having the same output (I explained here how it's done by calculating 2.5_factor).

So without giving you the trajectory information and right scale in meters, the model is capable of accurately estimating the human joint movements which can be useful for applications such as action recognition, etc.

[a-kungwani]

Thank you so much @SoroushMehraban. I really appreciate your detailed explanation and consistent replies on the issues.