Pose_to_Video not working

Question

Pose_to_Video not working

Closed this issue 10 months ago · 3 comments

Hey @AmitMY i have got the pose for my sentence "Manche Messungen haben nicht nur im März"

!text_to_gloss_to_pose \
  --text "Manche Messungen haben nicht nur im März" \
  --glosser "simple" \
  --lexicon "/content/drive/MyDrive/lexicon" \
  --spoken-language "de" \
  --signed-language "sgg" \
  --pose "quick_test.pose"

The pose to video module does'nt work for command

!pose_to_video --type pix2pix --model "/content/spoken-to-signed-translation/pix2pix.h5"  --pose "/content/spoken-to-signed-translation/quick_test.pose" --video "/content/spoken-to-signed-translation/quick_test.mp4" --upscale

It reports error

Loading input pose ...
2023-12-01 16:52:22.573597: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-12-01 16:52:22.573665: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-12-01 16:52:22.573702: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-12-01 16:52:23.694603: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Generating video ...
Traceback (most recent call last):
File "/usr/local/bin/pose_to_video", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/pose_to_video/init.py", line 36, in main
module = importlib.import_module(f"pose_to_video.{args.type}")
File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pose_to_video.pix2pix'

Answer 1 · 2023-12-02T08:59:14.000Z

You are correct. I migrated the pose-to-video component's code to https://github.com/sign-language-processing/pose-to-video (you'll have to pip install that repository at the moment, though it is not tested)
I will probably fix all of this mess once the controlnet model finishes training, since it is much better than the pix2pix although a lot slower.
Here's a training example (from the beginning of training)

Answer 2 · 2023-12-02T17:16:59.000Z

Thanks for the response !!
I tried pip installing the repo https://github.com/sign-language-processing/pose-to-video and pip install .[pix2pix]

It doesnt work for the command

!pose_to_video --type=pix2pix --model=pix_to_pix/training/model.h5 --pose=/content/spoken-to-signed-translation/quick_test.pose --video=sign.mp4

Error is

Loading input pose ...
2023-12-02 17:11:08.599346: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-12-02 17:11:08.599426: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-12-02 17:11:08.599470: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-12-02 17:11:08.612978: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-02 17:11:11.624834: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Generating video ...
pose_to_video.conditional.pix2pix
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pose_to_video/bin.py", line 39, in main
    module = importlib.import_module(f"pose_to_video.conditional.{args.type}")
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pose_to_video.conditional'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pose_to_video", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/pose_to_video/bin.py", line 41, in main
    module = importlib.import_module(f"pose_to_video.unconditional.{args.type}")
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pose_to_video.unconditional'

Even the example usage in https://github.com/sign-language-processing/pose-to-video reports the same error.

Is there something i am doing wrong?? @AmitMY

Answer 3 · 2024-02-12T13:44:55.000Z

My appologies it took long to address your issue.
I believe I just addressed your issue in sign-language-processing/pose-to-video@b40033b
Feel free to open an issue with that repository if you are having trouble with it.