dlopen libnvcuvid.so failed!. Please install GPU dirver
Closed this issue · 2 comments
robert-s-lee commented
Dali is failing with dlopen libnvcuvid.so failed
[root.my_work] 2022-06-26T01:59:14.586Z INFO: Initializing a HeatmapTracker instance.
[root.my_work] 2022-06-26T01:59:14.587Z INFO: [2022-06-26 01:59:14,586][py.warnings][WARNING] - /content/venv/lib/python3.8/site-packages/pytorch_lightning/core/saving.py:217: UserWarning: Found keys that are not in the model state dict but in the checkpoint: ['total_unsupervised_importance']
[root.my_work] 2022-06-26T01:59:14.587Z INFO: rank_zero_warn(
[root.my_work] 2022-06-26T01:59:14.587Z INFO:
[root.my_work] 2022-06-26T01:59:17.986Z INFO: dlopen libnvcuvid.so failed!. Please install GPU dirver------------------------------------
[root.my_work] 2022-06-26T01:59:17.986Z INFO: Building DALI video eval pipeline...
[root.my_work] 2022-06-26T01:59:17.986Z INFO: ------------------------------------
[root.my_work] 2022-06-26T01:59:17.986Z INFO: Error executing job with overrides: ['eval.hydra_paths=[2022-06-26/01-58-45]', 'eval.test_videos_directory=/content/lightning-pose/toy_datasets/toymouseRunningData/unlabeled_videos', 'eval.saved_vid_preds_dir=outputs/2022-06-26/01-58-45']
[root.my_work] 2022-06-26T01:59:17.991Z INFO: Traceback (most recent call last):
[root.my_work] 2022-06-26T01:59:17.991Z INFO: File "scripts/predict_new_vids.py", line 106, in <module>
[root.my_work] 2022-06-26T01:59:17.991Z INFO: predict_videos_in_dir()
[root.my_work] 2022-06-26T01:59:17.991Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/main.py", line 90, in decorated_main
[root.my_work] 2022-06-26T01:59:17.991Z INFO: _run_hydra(
[root.my_work] 2022-06-26T01:59:17.992Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra
[root.my_work] 2022-06-26T01:59:17.992Z INFO: _run_app(
[root.my_work] 2022-06-26T01:59:17.992Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 452, in _run_app
[root.my_work] 2022-06-26T01:59:17.992Z INFO: run_and_report(
[root.my_work] 2022-06-26T01:59:17.992Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 216, in run_and_report
[root.my_work] 2022-06-26T01:59:17.992Z INFO: raise ex
[root.my_work] 2022-06-26T01:59:17.992Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
[root.my_work] 2022-06-26T01:59:17.992Z INFO: return func()
[root.my_work] 2022-06-26T01:59:17.992Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 453, in <lambda>
[root.my_work] 2022-06-26T01:59:17.992Z INFO: lambda: hydra.run(
[root.my_work] 2022-06-26T01:59:17.992Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
[root.my_work] 2022-06-26T01:59:17.993Z INFO: _ = ret.return_value
[root.my_work] 2022-06-26T01:59:17.993Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
[root.my_work] 2022-06-26T01:59:17.993Z INFO: raise self._return_value
[root.my_work] 2022-06-26T01:59:17.993Z INFO: File "/content/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
[root.my_work] 2022-06-26T01:59:17.993Z INFO: ret.return_value = task_function(task_cfg)
[root.my_work] 2022-06-26T01:59:17.993Z INFO: File "scripts/predict_new_vids.py", line 81, in predict_videos_in_dir
[root.my_work] 2022-06-26T01:59:17.993Z INFO: preds_df, heatmaps_np = predict_single_video(
[root.my_work] 2022-06-26T01:59:17.993Z INFO: File "/content/venv/lib/python3.8/site-packages/typeguard/__init__.py", line 1033, in wrapper
[root.my_work] 2022-06-26T01:59:17.993Z INFO: retval = func(*args, **kwargs)
[root.my_work] 2022-06-26T01:59:17.994Z INFO: File "/content/lightning-pose/lightning_pose/utils/predictions.py", line 217, in predict_single_video
[root.my_work] 2022-06-26T01:59:17.994Z INFO: predict_loader = LightningWrapper(
[root.my_work] 2022-06-26T01:59:17.994Z INFO: File "/content/lightning-pose/lightning_pose/data/dali.py", line 96, in __init__
[root.my_work] 2022-06-26T01:59:17.994Z INFO: super().__init__(*kargs, **kwargs)
[root.my_work] 2022-06-26T01:59:17.994Z INFO: File "/content/venv/lib/python3.8/site-packages/nvidia/dali/plugin/pytorch.py", line 165, in __init__
[root.my_work] 2022-06-26T01:59:17.994Z INFO: _DaliBaseIterator.__init__(self,
[root.my_work] 2022-06-26T01:59:17.994Z INFO: File "/content/venv/lib/python3.8/site-packages/nvidia/dali/plugin/base_iterator.py", line 175, in __init__
[root.my_work] 2022-06-26T01:59:17.994Z INFO: p.build()
[root.my_work] 2022-06-26T01:59:17.994Z INFO: File "/content/venv/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 760, in build
[root.my_work] 2022-06-26T01:59:17.994Z INFO: self._pipe.Build(self._generate_build_args())
[root.my_work] 2022-06-26T01:59:17.995Z INFO: RuntimeError: Critical error when building pipeline:
[root.my_work] 2022-06-26T01:59:17.995Z INFO: Error when constructing operator: readers__Video encountered:
[root.my_work] 2022-06-26T01:59:17.995Z INFO: [/opt/dali/dali/operators/reader/loader/video_loader.h:191] Assert on "ret" failed: Failed to load libnvcuvid.so, needed by the VideoReader operator. If you are running in a Docker container, please refer to https://github.com/NVIDIA/nvidia-docker/wiki/Usage
[root.my_work] 2022-06-26T01:59:17.995Z INFO: Stacktrace (70 entries):
robert-s-lee commented
Nvidia's remedy is to add
So try to add to your nvidia-docker/docker command following option --gpus '"capabilities=compute,utility,video"' as described here.
[root.my_work] 2022-06-26T01:59:17.986Z INFO: dlopen libnvcuvid.so failed!. Please install GPU dirver------------------------------------
[root.my_work] 2022-06-26T01:59:17.986Z INFO: Building DALI video eval pipeline...
[root.my_work] 2022-06-26T01:59:17.986Z INFO: ------------------------------------
robert-s-lee commented
fixed via lightning run app app.py --cloud --env NVIDIA_DRIVER_CAPABILITIES=compute,utility,video