cceyda/image-checker

`Critical error when building pipeline`

strarsis opened this issue · 1 comments

image-checker
{'path': './', 'batch_size': 50, 'device_id': 0, 'extensions': ['jpeg', 'jpg', 'png'], 'log_conf': 'logging_config.json', 'recursive': False, 'debug': False, 'use_cpu': False, 'prefetch': 1, 'device': 'mixed'}
Good images (±50): 0it [00:00,/usr/local/lib/python3.8/dist-packages/nvidia/dali/ops.py:649: DeprecationWarning: WARNING: `image_decoder` is now deprecated. Use `decoders.image` instead.
In DALI 1.0 all decoders were moved into a dedicated :mod:`~nvidia.dali.fn.decoders`
submodule and renamed to follow a common pattern. This is a placeholder operator with identical
functionality to allow for backward compatibility.
  op_instances.append(_OperatorInstance(input_set, self, **kwargs))
Traceback (most recent call last):
  File "/home/build/.local/bin/image-checker", line 8, in <module>
    sys.exit(main())
  File "/home/build/.local/lib/python3.8/site-packages/image_checker/cli.py", line 73, in main
    checker_batch(ds, args)
  File "/home/build/.local/lib/python3.8/site-packages/image_checker/checker.py", line 38, in checker_batch
    checker = DaliChecker(args["batch_size"], args["prefetch"],args["device"],args["device_id"])
  File "/home/build/.local/lib/python3.8/site-packages/image_checker/DaliChecker.py", line 27, in __init__
    self.pipe.build()
  File "/usr/local/lib/python3.8/dist-packages/nvidia/dali/pipeline.py", line 828, in build
    self._pipe.Build(self._generate_build_args())
RuntimeError: Critical error when building pipeline:
Error when constructing operator: ImageDecoder encountered:
Error in thread 0: nvml error (3): The nvml requested operation is not available on target device
Current pipeline object is no longer valid.
Bad images: 0it [00:02, ?it/s]
Good images (±50): 0it [00:02, ?it/s]

This runs on WSL 2 with NVIDIA drivers and CUDA 11 (11.0.228) support installed and available.
Or does Dali require explicit JPEG/image hardware decoder support on the GPU, which isn't present or passed-through on this system?

Apparently this is an issue with the current NVML support on WSL 2. Disabling NVML for DALI by setting the environment variable DALI_DISABLE_NVML=1 fixes the issue.

DALI_DISABLE_NVML=1 image-checker

(Or use export DALI_DISABLE_NVML=1 or add DALI_DISABLE_NVML=1 to your e.g. .bashrc).