could the network be trained with only action video but no pose images/video?

[ardeal]

Hi,

I am working on the following scenario:
The camera is hung on the ceiling, and monitoring the person who is working on the table with his hands. The software is going to recognition the action of the person's hand. I would like to recognize/classify normal action and abnormal action.

as you mentioned in your readme.md, I got a few questions:

• 1. You trained the network with MPII(2D pose), Human3.6(3D pose) and NTU(action). however, I have only action video shot from the ceiling, but I don't have 2D pose nor 3D pose. Could I trained the network with my video/image sequence? and how to train it?
• 2. according to your experience, could my action be correctly recognized by the model trained by my video? how about the accuracy? please give an estimation.
• 3. how much training images/videos are needed for my case?
• 4. in your readme.md, you didn't write down your email address. :)

Multi-task model for 3D pose estimation and action recognition

This model was trained simultaneously on MPII (2D pose), Human3.6 (3D pose) and NTU (action), and the results can be replicated on NTU for action recognition by:

  python3 exp/ntu/eval_ntu_multitask.py output/eval-ntu
```C


Thanks,
Ardeal

[dluvizon]

Hi @ardeal ,

Thank you for your interest. Here are my comments.

i. For this question, please check our Table 6 in our TPAMI paper. You can train only action, but the scores are lower.
ii. I don't know, it depends on how difficult is your scenario. You'll have to try it.
iii. Similarly to ii, it depends on the videos/images that you have in our scenario.
iv. You can find my e-mail in my personal web page.

Best,
Diogo

[ardeal]

Hi Diogo,

Many thanks for your reply!

for my scenario, I have the human action videos, and I could label all videos or split the action to separate frames for training. if I would like to train 2D pose and and 3D pose, should I use MPII 2D pose images and Human3.6 3D pose images? or should I create new 2D and 3D pose images?

for the training, I have no idea about the steps, could you please help to see whether the following understanding is correct or not?

1. input many batches(4 or 8 or 16 images or etc.) of 2D pose images to the network to train.
2. input many batches(4 or 8 or 16 images or etc.) of 3D pose images to the network to train.
3. separate the action videos to many frames(such as 8, or 9 or other number images), and input those frames separated from 1 action to train. and then continue with the next action video.

Thanks,
Ardeal

[dluvizon]

Hi,

Maybe in your case you'll need to collect or use some other datasets than MPII or H36M, since these datasets have the camera usually in front of the people.
For the training process, you can input as many images in the batch as you can (depending on your GPU memory). For each image that you have, you can supervise the network with the labels that you have, either 2D, 3D poses and action labels (for a sequence of frames). If you don't have a specific label for a given sequence of frames, you propagate zero in the respective loss function.

Best,

[github-actions]

Stale issue message