Test a video from youtube or test my picture?
Closed this issue · 44 comments
dear @una-dinosauria ,
Can you please me how to test a video from youtube or test my picture after finishing training?
Check out this repo: Arash Hosseini's repo, which does exactly what you want: https://github.com/ArashHosseini/3d-pose-baseline
download your YouTube Video, make sure there are not digits in the file Name. Use ffmpeg to create a Image Sequenze from them. Parse the Images into your selected 2d-Pose-estimation system like openpose. The Output of this, the json Files are now ready to be use by 3D-Pose-baseline.
@ArashHosseini , will this work also for posenet with tensor flow.js ? Do you know or have idea how the performance is compared to open pose. I had tried to do 2d pose estimation on an youtube video using ml5 java script wrapper but the performance was quite bad . Your thoughts ?
@ArashHosseini I have done this. I get good results from 2d-pose-estimation system (openpose) but when I move into 3d-pose-estimation the stick figure isn't even close to the way my 2d-keypoints where.
I run python src/openpose_3dpose_sandbox.py --camera_frame --residual --batch_norm --dropout 0.5 --max_norm --evaluateActionWise --use_sh --epochs 200 --load 4874200 --pose_estimation_json /path/to/json_directory --write_gif --gif_fps 24 as suggested. Do you have any suggestions for how I could improve the results?
@basicvisual, good point, not yet. is posenet writing any file down, i saw its returning the keypoints, have you done something about writing json files, maybe here? If not i can also take the returned kepoints in the example to add posenet also to the pose-estimators list, let it me know.
@Shoulder95, can you give us an example of your json files? thx
@ArashHosseini Here is one example
{"people": [{"pose_keypoints_2d": [374, 460, 374, 516, 324, 518, 296, 596, 336, 636, 424, 512, 446, 590, 424, 604, 340, 660, 324, 776, 308, 890, 400, 660, 402, 792, 400, 904, 364, 448, 382, 450, 348, 450, 396, 450]}]}
@Shoulder95, i will check it. thx
@ArashHosseini I actually used the ml5 javascript wrapper. I actually posted the question in stack exchange. I have the code (basic one) for which i tried to do open pose and corresponding link to it .
I am not trying real time pose estimation with webcamera , but rather a video stored locally. Apart from below nothing much has been done . Was just exploring the wrapper .
@Shoulder95, that is the pose from your json, is your output looks different then that?
@ArashHosseini I get the same as you. The true pose is much straighter.
Here is the result I get from tf-openpose.
@Shoulder95, ok i didn't have the ground truth, i see. its in all images like this?
However , the output skeleton seems to be quite off
i found drawPoint(ctx, y * scale, x * scale, 3, color); here, the scale factor could be responsible for this behavior, set scale to 1 and try it again please, i found also this "The output stride and image scale factor have the largest effects on accuracy/speed. A higher output stride results in lower accuracy but higher speed. A higher image scale factor results in higher accuracy but lower speed. " here
@ArashHosseini yes, I get a hunched over looking figure in most figures. I have also tried running a larger portion of the video with --interpolation and --multiplier 1, to some minor improvement but not close to the results you display in the repo.
Do you think it could be a scale problem? Maybe I should try cropping the video, but I feel like that should also affect the json 2d-identification.
Thank you for all the help!
@ArashHosseini , while browsing through issues , related to pose-estimation with posenet . I found the following porting of posenet to python. Maybe this is off a good reference ? https://github.com/rwightman/posenet-python
dear @una-dinosauria,
I run it: https://github.com/ArashHosseini/tf-pose-estimation
This Step: " Tensorflow
fork tf-pose-estimation and add --output_json flag to export Pose Keypoints like python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /tmp/, check diff "
My command is that:
cd tf-openpose
python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /tmp/
My question is that: Why is my terminal run continuously and my camera run continuously?
How can i save file json?
Dear @una-dinosauria ,
Look the video below:
Screencast from 04-04-2019 00:15:09.zip
Hey @NguyenDangBinh, I hope I understand your question correctly
https://github.com/ArashHosseini/tf-pose-estimation/blob/9a8509815081bdc246830bb031429fdf0a651e3f/run_webcam.py#L52 is using your webcam image to predict the pose and saving the output prediction into json files, in your case in /tmp/ folder, take a look inside it.
Why is my terminal run continuously and my camera run continuously?
that's why your camera runs continuously and tf-pose-estimation shows the results in the terminal continuously, I hope that answers your question.
dear @una-dinosauria,
I understood the reason the camera run continuously. But How can i save the json file? Because in /tmp/ folder, I did not see any files.
@ArashHosseini I havent managed to export , to a json file. I can definetly see the keypoints in a output console but not a top down approach. Basically using the ml5wrapper
@basicvisual, good point, not yet. is posenet writing any file down, i saw its returning the keypoints, have you done something about writing json files, maybe here? If not i can also take the returned kepoints in the example to add posenet also to the pose-estimators list, let it me know.
@basicvisual it would be awesome if we could manage the export part, i am not the best in javascript, but will try to add json writing for the demo, we need somehow the output from the console as json
dear @ArashHosseini,
If I run below:
python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /path/to/directory
Can you show me how to save json file because the Teminal run continuously?
Thank you
hey @NguyenDangBinh, run python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /home/ and interrupt the terminal manually, now every frame from your webcam should create in /home/ a json with frame name like 000000000001.json. let it me know if you can see the json files. thx
@ArashHosseini , unfortunately with the current implmentation of posenet in tensorflow js, it is not possible to export the keypoints to json as it runs , in the browser so we cannot actually use the commands such as fswritefile for java script and the tensorflow model has not been completely ported to nodejs.
However, in the discussions, some one pointed to python implementation of posenet model
https://github.com/zg9uagfv/tf_posenet
I found the variable instanceKeypoints in decode_pose.py which returns the keypoints. I have not managed to export to json yet. I will see , if the community can solve the other issue .
dear @ArashHosseini,
If I run:
(general) binh@binh-Z440:~/tf-openpose$ python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /home/
then it has error
Traceback (most recent call last):
File "run_webcam.py", line 63, in
image = TfPoseEstimator.draw_humans(image, humans, imgcopy=False, frame=frame, output_json_dir=args.output_json)
File "/home/binh/tf-openpose/tf_pose/estimator.py", line 433, in draw_humans
with open(os.path.join(output_json_dir, '{0}_keypoints.json'.format(str(frame).zfill(12))), 'w') as outfile:
PermissionError: [Errno 13] Permission denied: '/home/000000000000_keypoints.json'
Hi @NguyenDangBinh, not bad, try it with cd /home/ following by sudo mkdir openpose_output, then cd ~/tf-openpose and python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /home/openpose_output
Hey @basicvisual thanks for the note, check diff, thx
dear @ArashHosseini
(general) binh@binh-Z440:~/tf-openpose$ python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /home/openpose_output
[2019-04-07 21:23:51,858] [TfPoseEstimator-WebCam] [DEBUG] initialization mobilenet_thin : /home/binh/tf-openpose/models/graph/mobilenet_thin/graph_opt.pb
[2019-04-07 21:23:51,859] [TfPoseEstimator] [INFO] loading graph from /home/binh/tf-openpose/models/graph/mobilenet_thin/graph_opt.pb(default size=432x368)
2019-04-07 21:23:51.921737: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-04-07 21:23:51.943325: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3491965000 Hz
2019-04-07 21:23:51.943922: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x563213a21030 executing computations on platform Host. Devices:
2019-04-07 21:23:51.943942: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-04-07 21:23:52.047958: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5632126df350 executing computations on platform CUDA. Devices:
2019-04-07 21:23:52.047999: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
2019-04-07 21:23:52.048751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:02:00.0
totalMemory: 7.93GiB freeMemory: 7.42GiB
2019-04-07 21:23:52.048780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-04-07 21:23:52.050141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-07 21:23:52.050163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-04-07 21:23:52.050172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-04-07 21:23:52.050854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7223 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:02:00.0, compute capability: 6.1)
WARNING:tensorflow:From /home/binh/anaconda3/envs/general/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[2019-04-07 21:23:53,999] [TfPoseEstimator-WebCam] [DEBUG] cam read+
[2019-04-07 21:23:56,381] [TfPoseEstimator-WebCam] [INFO] cam image=640x480
[2019-04-07 21:23:56,412] [TfPoseEstimator-WebCam] [DEBUG] image process+
[2019-04-07 21:23:56,447] [TfPoseEstimator-WebCam] [DEBUG] postprocess+
Traceback (most recent call last):
File "run_webcam.py", line 63, in
image = TfPoseEstimator.draw_humans(image, humans, imgcopy=False, frame=frame, output_json_dir=args.output_json)
File "/home/binh/tf-openpose/tf_pose/estimator.py", line 433, in draw_humans
with open(os.path.join(output_json_dir, '{0}_keypoints.json'.format(str(frame).zfill(12))), 'w') as outfile:
PermissionError: [Errno 13] Permission denied: '/home/openpose_output/000000000000_keypoints.json'
The similar error.
@NguyenDangBinh ok, i hopped the creating of the folder will solve it but the basic problem is that home is not own by you, because of this Permission denied. see possible fix
dear @ArashHosseini,
I run (general) binh@binh-Z440:~/tf-openpose$ python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /tmp/
successfully.
But I can not see any output_json file in tmp folder. Why?
@NguyenDangBinh i have no idea, Do you show the whole body?
dear @ArashHosseini,
Step 1: I run: (general) binh@binh-Z440:~/tf-openpose$ python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0 --output_json /tmp/
Step 2: press ESP to finish: I change this code on run_camera.py like:
if cv2.waitKey(33) ==27:
break
But I do not see any file Json in tmp folder.
See video record below.
Screencast from 07-04-2019 23:04:.zip
@ArashHosseini , i did some modifications to run the modified code. Basically giving absolute path in draw.py in the following
def drawKeypoints(body, img, color, frame=0,output="/home/ubuntu/tf_posenet"): and also in the file posenet.py . The json file outputted is called 000000000000_keypoints right ? The sample output looks as follows for the json file
{"people": [{"pose_keypoints_2d": [255.01425446530124, 52.52263451188538, 259.33765912692695, 48.79566056792036, 250.4453495546335, 49.23223569060646, 266.68261553908667, 51.388602440718024, 244.24452637564886, 52.41461552922733, 276.03850270449584, 73.38322930944426, 239.99023126211645, 74.12114265057029, 291.7458303179868, 95.9702590806548, 232.07244987657586, 98.62804718470362, 279.6036232062546, 99.23267828604588, 224.1749938093236, 102.08677934256082, 271.63201709883145, 120.74003304501316, 249.03411605167105, 121.76984204029118, 274.4318254736835, 158.96242385094527, 251.83990930449713, 158.73652562897, 278.95295081718734, 191.59205745165002, 268.5772169973447, 190.8127650592023, 0.0, 0.0]}]}
I am not sure how your results are but , for me they were not the greatest. I think there must be a lot of tweaking done in the code. I am running on a Virtual Machine which has access to the Nvdia graphics card quadro M2200 . I attach a few screenshots . Please let me know how it looks for you ? Also , whenever the pose-estimation is done on a newer video or different video , i guess the json file is overwritten .
Link is provided below
https://drive.google.com/open?id=1ULkYkFaE8AyQC6pPGys0vj36JFHnizo5
Hi @basicvisual, looks good
The json file outputted is called 000000000000_keypoints right
correct, can you please make a prediction with
and uploading the json please? thx
@ArashHosseini , I have to learn how i can do that . I have not done that part . And i am not sure what exactly needs to be done . :) Do you mean to upload in the google drive ? ( i have uploaded one )
@ArashHosseini , i was trying to input the json file to openpose_3dpose_sandbox.py to do the 3d pose estimation . I was wondering if that is the correct way to do so ? Could you had a time to see the json file ? I guess the code in openpose_3dpose_sandbox.py has to be modified ? Also , there is a possibility to save output console log from the browser into json ( using method blob ) . I will look into it , Please let me know your thoughts .
dear @ArashHosseini,
Do you remember my issues?
@NguyenDangBinh does your tmp folder have read /write permissions ?
dear @basicvisual,
I check that:
(general) binh@binh-Z440:~/tf-openpose$ ls -ld tmp
drwxr-xr-x 2 binh binh 4096 Thg 4 7 21:26 tmp
I think tmp folder had read/write permission?
Hello @ArashHosseini , after playing around , I had a better understanding of the whole structure and code . I have a following set of questions
-
If i understand the
output_jsonis kind doing some kind of buffering ? Is it possible to save the entire 2D detections of the poses in ajsonfile ? -
Is it possible to read out the 3D poses to a text file ? ( the entire sequence ie from the start to end of the video ?
-
For my case I started the 2D detection module first and then fed in the
jsonoutput in real time and ran the following commandpython3.5 src/openpose_3dpose_sandbox_realtime.py --camera_frame --residual --batch_norm --dropout 0.5 --max_norm --evaluateActionWise --use_sh --epochs 200 --load 4874200 --pose_estimation_json /home/tm/prot/tf_posenet/ --verbose 3? -
I dont get that good performance though for 3D pose estimation. I am however running it on a PC without GPU support. Could that effect the performance. I notice if the object faces the camera, the 2D detections are good .(file
Trialonerealtimebut as soon the subject is facing side ways the 2D detection is just off (fileTrialonerealtime2)
Link to the video files https://drive.google.com/open?id=1ygsnZ8SncUJdSE_yFPfzXPBf_aEzBZkq -
I read some where in threads , that you must download the pre trained model
exportto the top level and then run theposeestimationsandbox
I would like to hear your views on the above questions and also how does your performance looks like ?
@NguyenDangBinh a simple check if you can create some files and copy to that folder. If you can then you have permission.
dear @basicvisual,
I can create folder in that.
@ArashHosseini , I have created another issue regarding , posenet-python. I have added some code but i just cannot manage to output to dump the keypoints co-ordinates to json file and format, that you have specified .
link is here
Closing for lack of activity. Please reopen if the issue is still ongoing.
@ArashHosseini , could you confirm this with the following issue
#155