michalfaber/keras_Realtime_Multi-Person_Pose_Estimation

Anyone know how to get specific body part?

oakkas opened this issue · 6 comments

oakkas commented

Hi all,
Thanks for the great work.
I am working on estimating specific body posture (shoulder flexion, extension). I also want to estimate angles. Do anyone know how to extract these from the code. I feel I can use key points but could not be sure which key points associated with which body parts.

Thanks in advance.

rludlow commented

Hi - the joints are coded as:

joint_codes = {
0: "head", 1: "neck" ,2: "r.shoulder", 3: "r.elbow", 4: "r.wrist", 5: "l.shoulder", 6: "l.elbow", 7: "l.wrist", 8: "r.hip", 9: "r.knee", 10: "r.ankle", 11: "l.hip", 12: "l.knee", 13: "l.ankle", 14: "l.eye", 15: "r.eye", 16: "l.ear", 17: "r.ear", 18: "pt19"
}

In demo_image.py and the demo notebook, you can get the coordinates and confidence score for each joint with all_peaks[num] and the heatmaps with heatmap_avg[num].

oakkas commented

@rludlow Thanks for your reply. In all_peaks for each row I sometimes get two arrays and sometimes one. Do you have any idea why. Also what is the difference between all_peaks and candidate?
Also can you please explain what "subset" is for. I realize that for an image I use, sum of the indexes are excluded. The omitted indexes seem like related to an extra array in "all_peaks". Are these different estimations of a joint?

Thanks

rludlow commented

Thanks for your question. Working through this has helped me better understand the program. First I'll mention that running the demo on an image with two people might make it easier to understand what's going on. But here's what I see:

  • all_peaks contains all the joint coordinates (and score, and assigned id) found for each body part. You see two or more arrays when there are two or more of a given joint found; so if all_peaks[3] = [(459, 447, 0.8317168653011322, 13), (1095, 455, 0.83217310905456543, 14)], then the system thinks it sees right elbows for two different people. Note that sometimes it will find a random joint erroneously, so you'll have an unexpected double entry.

  • candidate just lists all of the entries from all_peaks, giving each entry [x, y, score, id] its own line, rather than grouping all entries for a given body part in a combined array.

  • subset is essentially an array of all the suspected people in the image. subset[i] entries 0-18 are the id's from all_peaks being associated with person[i] for each body part 0-18 (it will have a -1 if the part if not found for that person); entry 19 is the overall score for the person, and entry 20 is the number of found parts for that person.

I'm not totally sure what you mean when you say some indices are excluded. Possible explanations are that a joint wasn't found in your image (it doesn't accurately identify all of them all the time), or that a joint was found and included in all_peaks but was excluded, for a reason such as it belonged to a "person" that didn't have enough identified parts to be included. The array deleteIdx includes those removed for this reason.

Again, doing it with two people helps clarify things. When I ran it with one, the extra entries in all_peaks just seemed like random errors. With two, you see how the program is dealing with finding multiple entries for a given body_part.

  • 👍2
oakkas commented

Thanks for your suggestion. I will try with an image containing two people. Hope it will help me better understand along with your comments. Best.

Ai-is-light commented

Thanks for your question. Working through this has helped me better understand the program. First I'll mention that running the demo on an image with two people might make it easier to understand what's going on. But here's what I see:

  • all_peaks contains all the joint coordinates (and score, and assigned id) found for each body part. You see two or more arrays when there are two or more of a given joint found; so if all_peaks[3] = [(459, 447, 0.8317168653011322, 13), (1095, 455, 0.83217310905456543, 14)], then the system thinks it sees right elbows for two different people. Note that sometimes it will find a random joint erroneously, so you'll have an unexpected double entry.
  • candidate just lists all of the entries from all_peaks, giving each entry [x, y, score, id] its own line, rather than grouping all entries for a given body part in a combined array.
  • subset is essentially an array of all the suspected people in the image. subset[i] entries 0-18 are the id's from all_peaks being associated with person[i] for each body part 0-18 (it will have a -1 if the part if not found for that person); entry 19 is the overall score for the person, and entry 20 is the number of found parts for that person.

I'm not totally sure what you mean when you say some indices are excluded. Possible explanations are that a joint wasn't found in your image (it doesn't accurately identify all of them all the time), or that a joint was found and included in all_peaks but was excluded, for a reason such as it belonged to a "person" that didn't have enough identified parts to be included. The array deleteIdx includes those removed for this reason.

Again, doing it with two people helps clarify things. When I ran it with one, the extra entries in all_peaks just seemed like random errors. With two, you see how the program is dealing with finding multiple entries for a given body_part.

@rludlow each body part 0-18 (it will have a -1 if the part if not found for that person), there should be 0-17.
And I have one question about " entry 19 is the overall score for the person"(there maybe also entry18), I test some images and found the the score maybe > 18*1, like 28.431490042423025，25.830770298037518，so, how to compute the score? The score is total keypoint confidence of that person, even though some keypoint have been deleted, which confidence score have been computed?
Thx

rafikg commented

@rludlow Thanks for the explanation.
I have compared between the cmu model implemented here and mobilenet_thin implemented in this githubopenpose.
For the prediction time with same size of the image and I see that as expected the prediction time of Mobilenet_thin is much faster (5 time faster for the same image size). However, the bottleneck is the post_processing part specially when I increase the size of the heatmap, the prediction time is still small but the extract_parts() takes too much time!