Data Read Error During Evaluation
Closed this issue · 20 comments
ztrbq commented
Thanks for your work magehrig.
I get an error when I try to run the validation file and the error shows as this:
---------------------------
Using 16bit native Automatic Mixed Precision (AMP)
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.model_summary.ModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
creating streaming test datasets: 470it [00:00, 755.23it/s]
num_full_sequences=470
num_splits=0
num_split_sequences=0
Restoring states from the checkpoint path at rvt-t.ckpt
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Loaded model weights from checkpoint at rvt-t.ckpt
/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:224: PossibleUserWarning: The dataloader, test_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 20 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
Testing DataLoader 0: : 0it [00:00, ?it/s]/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1678402412426/work/aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Testing DataLoader 0: : 735it [03:09, 3.88it/s]Error executing job with overrides: ['dataset=gen1', 'dataset.path=../Gen1/gen1', 'checkpoint=./rvt-t.ckpt', 'use_test_set=1', 'hardware.gpus=0', '+experiment/gen1=tiny.yaml', 'batch_size.eval=8', 'model.postprocess.confidence_threshold=0.001']
Traceback (most recent call last):
File "/home/zbq/Desktop/RVT-master/validation.py", line 84, in main
trainer.test(model=module, datamodule=data_module, ckpt_path=str(ckpt_path))
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 780, in test
return call._call_and_handle_interrupt(
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 829, in _test_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1098, in _run
results = self._run_stage()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1174, in _run_stage
return self._run_evaluate()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_evaluate
eval_loop_results = self._evaluation_loop.run()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 121, in advance
batch = next(data_fetcher)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in __next__
return self.fetching_function()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 258, in fetching_function
self._fetch_next_batch(self.dataloader_iter)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 280, in _fetch_next_batch
batch = next(iterator)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 634, in __next__
data = self._next_data()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data
return self._process_data(data)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data
data.reraise()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/_utils.py", line 644, in reraise
raise exception
OSError: Caught OSError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 41, in fetch
data = next(self.dataset_iter)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 144, in __next__
return self._get_next()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 132, in _get_next
result = next(self.iterator)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 215, in wrap_next
result = next_func(*args, **kwargs)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/datapipe.py", line 369, in __next__
return next(self._datapipe_iter)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 144, in __next__
return self._get_next()
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 132, in _get_next
result = next(self.iterator)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 185, in wrap_generator
response = gen.send(request)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/iter/combining.py", line 589, in __iter__
yield from zip(*iterators)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 185, in wrap_generator
response = gen.send(request)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torchdata/datapipes/iter/util/zip_longest.py", line 56, in __iter__
value = next(iterators[i])
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 185, in wrap_generator
response = gen.send(request)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/iter/combining.py", line 52, in __iter__
for data in dp:
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 185, in wrap_generator
response = gen.send(request)
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torchdata/datapipes/map/util/converter.py", line 47, in __iter__
yield self.datapipe[idx]
File "/home/zbq/Desktop/RVT-master/data/genx_utils/sequence_for_streaming.py", line 152, in __getitem__
ev_repr = self._get_event_repr_torch(start_idx=start_idx, end_idx=end_idx)
File "/home/zbq/Desktop/RVT-master/data/genx_utils/sequence_base.py", line 91, in _get_event_repr_torch
ev_repr = h5f['data'][start_idx:end_idx]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/h5py/_hl/dataset.py", line 768, in __getitem__
return self._fast_reader.read(args)
File "h5py/_selector.pyx", line 376, in h5py._selector.Reader.read
OSError: Can't synchronously read data (Blosc decompression error)
This exception is thrown by __iter__ of MapToIterConverterIterDataPipe(datapipe=SequenceForIter, indices=range(0, 57))
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
== Timing statistics ==
Testing DataLoader 0: : 735it [03:09, 3.87it/s]
I refer to your previous work issue and tried to fix it through reinstall the hdf5plugin but it doesn't work.
I follow the instruction and use mamba instead of conda to install the package. And my system is Ubuntu 22.04 LTS.
robust1997 commented
I met the same error. But the strange thing is this error only appears on one of my device. For my server and other computers, this code works quite well, even though they have the same configuration as my computer.
magehrig commented
It's hard for me to fix this because I cannot reproduce it, unfortunately (I used 5 different machines and it never occurred on any of those).
Can you also post the output of conda list here?
-
A quick workaround is to set the number of workers for evaluation to "0" (that means using only the main process) by appending the following the args:
hardware.num_workers.eval=0 -
Potentially a solution to the problem is to
pip install hdf5plugin(you would probably have to remove theblosc-hdf5-pluginfrom the conda env) and then change the code to the following whenever h5py is imported (I have not tested this approach):
import hdf5plugin
os.environ["HDF5_PLUGIN_PATH"] = hdf5plugin.PLUGINS_PATH
import h5pyLet me know if any of this helps
robust1997 commented
Here is the output of my conda list:
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
absl-py 1.4.0 pypi_0 pypi
aiohttp 3.8.4 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
antlr-python-runtime 4.9.3 pyhd8ed1ab_1 conda-forge
appdirs 1.4.4 pypi_0 pypi
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
aws-c-auth 0.6.28 hccec9ca_5 conda-forge
aws-c-cal 0.5.27 hf85dbcb_0 conda-forge
aws-c-common 0.8.20 hd590300_0 conda-forge
aws-c-compression 0.2.17 h4b87b72_0 conda-forge
aws-c-event-stream 0.3.0 hc5de78f_6 conda-forge
aws-c-http 0.7.8 h412fb1b_4 conda-forge
aws-c-io 0.13.26 h0d05201_0 conda-forge
aws-c-mqtt 0.8.13 ha5d9b87_2 conda-forge
aws-c-s3 0.3.4 h95e21fb_5 conda-forge
aws-c-sdkutils 0.1.10 h4b87b72_0 conda-forge
aws-checksums 0.1.16 h4b87b72_0 conda-forge
aws-crt-cpp 0.20.2 h5289e1f_9 conda-forge
aws-sdk-cpp 1.10.57 h8101662_14 conda-forge
awscli 1.27.153 py39hf3d152e_0 conda-forge
backcall 0.2.0 pyhd3eb1b0_0
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd3eb1b0_0
bbox-visualizer 0.1.0 pypi_0 pypi
black 23.3.0 pypi_0 pypi
blas 1.0 mkl
blosc 1.21.4 h0f2a231_0 conda-forge
blosc-hdf5-plugin 1.0.0 h91a81c6_5 conda-forge
botocore 1.29.153 pyhd8ed1ab_0 conda-forge
brotlipy 0.7.0 py39hb9d737c_1005 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.19.1 hd590300_0 conda-forge
ca-certificates 2023.05.30 h06a4308_0
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 5.3.1 pypi_0 pypi
certifi 2023.5.7 py39h06a4308_0
cffi 1.15.1 py39he91dace_3 conda-forge
charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge
click 8.1.3 pypi_0 pypi
cloudpickle 2.2.1 pypi_0 pypi
colorama 0.4.4 pyhd3eb1b0_0
comm 0.1.3 pyhd8ed1ab_0 conda-forge
contourpy 1.1.0 pypi_0 pypi
cryptography 41.0.1 py39hd4f0224_0 conda-forge
cuda-cudart 11.8.89 0 nvidia
cuda-cupti 11.8.87 0 nvidia
cuda-libraries 11.8.0 0 nvidia
cuda-nvrtc 11.8.89 0 nvidia
cuda-nvtx 11.8.86 0 nvidia
cuda-runtime 11.8.0 0 nvidia
cycler 0.11.0 pypi_0 pypi
debugpy 1.6.7 py39h227be39_0 conda-forge
decorator 5.1.1 pyhd3eb1b0_0
detectron2 0.6 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
docutils 0.16 py39hf3d152e_3 conda-forge
einops 0.6.0 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.12.2 pyhd8ed1ab_0 conda-forge
fonttools 4.40.0 pypi_0 pypi
freetype 2.12.1 hca18f0e_1 conda-forge
frozenlist 1.3.3 pypi_0 pypi
fsspec 2023.6.0 pypi_0 pypi
fvcore 0.1.5.post20221221 pypi_0 pypi
gitdb 4.0.10 pypi_0 pypi
gitpython 3.1.31 pypi_0 pypi
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
google-auth 2.20.0 pypi_0 pypi
google-auth-oauthlib 1.0.0 pypi_0 pypi
grpcio 1.54.2 pypi_0 pypi
h5py 3.8.0 nompi_py39h89bf01e_101 conda-forge
hdf5 1.14.0 nompi_hb72d44e_103 conda-forge
hydra-core 1.3.2 pyhd8ed1ab_0 conda-forge
icu 72.1 hcb278e6_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.6.0 pypi_0 pypi
importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge
iopath 0.1.9 pypi_0 pypi
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.20.1 h81ceb04_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.15 hfd0df8a_0 conda-forge
ld_impl_linux-64 2.38 h1181459_1
lerc 4.0.0 h27087fc_0 conda-forge
libaec 1.0.6 hcb278e6_1 conda-forge
libcublas 11.11.3.6 0 nvidia
libcufft 10.9.0.58 0 nvidia
libcufile 1.6.1.9 0 nvidia
libcurand 10.3.2.106 0 nvidia
libcurl 8.1.2 h409715c_0 conda-forge
libcusolver 11.4.1.48 0 nvidia
libcusparse 11.7.5.86 0 nvidia
libdeflate 1.17 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.4 h6a678d5_0
libgcc-ng 13.1.0 he5830b7_0 conda-forge
libgfortran-ng 13.1.0 h69a702a_0 conda-forge
libgfortran5 13.1.0 h15d22d2_0 conda-forge
libhwloc 2.9.1 nocuda_h7313eea_6 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libllvm14 14.0.6 hcd5def8_3 conda-forge
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnpp 11.8.0.86 0 nvidia
libnvjpeg 11.9.0.86 0 nvidia
libpng 1.6.39 h753d276_0 conda-forge
libsodium 1.0.18 h7b6447c_0
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge
libtiff 4.5.0 h6adf6a1_2 conda-forge
libwebp-base 1.3.0 h0b41bf4_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.11.4 h0d562d8_0 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
lightning-utilities 0.8.0 pypi_0 pypi
llvm-openmp 16.0.5 h4dfa4b3_0 conda-forge
llvmlite 0.40.0 py39h174d805_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
markdown 3.4.3 pypi_0 pypi
markupsafe 2.1.3 py39hd1e30aa_0 conda-forge
matplotlib 3.7.1 pypi_0 pypi
mkl 2021.4.0 h8d4b97c_729 conda-forge
mkl-service 2.4.0 py39h7e14d7c_0 conda-forge
mkl_fft 1.3.1 py39h0c7bc48_1 conda-forge
mkl_random 1.2.2 py39hde0f152_0 conda-forge
mpmath 1.3.0 pyhd8ed1ab_0 conda-forge
multidict 6.0.4 pypi_0 pypi
mypy-extensions 1.0.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.5.6 py39h06a4308_0
nettle 3.6 he412f7d_0 conda-forge
networkx 3.1 pyhd8ed1ab_0 conda-forge
numba 0.57.0 py39hb75a051_1 conda-forge
numpy 1.24.3 py39h14f4228_0
numpy-base 1.24.3 py39h31eccc5_0
oauthlib 3.2.2 pypi_0 pypi
omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge
opencv-python 4.6.0.66 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.1.1 hd590300_1 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pandas 1.5.3 pypi_0 pypi
parso 0.8.3 pyhd3eb1b0_0
pathspec 0.11.1 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
pillow 9.4.0 py39h2320bf1_1 conda-forge
pip 23.1.2 py39h06a4308_0
platformdirs 3.5.3 pypi_0 pypi
plotly 5.13.1 pypi_0 pypi
portalocker 2.7.0 py39hf3d152e_0 conda-forge
prompt-toolkit 3.0.36 py39h06a4308_0
protobuf 3.20.3 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3eb1b0_2
pure_eval 0.2.2 pyhd3eb1b0_0
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.3.0 pypi_0 pypi
pycocotools 2.0.6 pypi_0 pypi
pycparser 2.21 pyhd3eb1b0_0
pygments 2.15.1 py39h06a4308_1
pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.0.9 pypi_0 pypi
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.9.16 h955ad1f_3
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytorch 2.0.0 py3.9_cuda11.8_cudnn8.7.0_0 pytorch
pytorch-cuda 11.8 h7e8668a_5 pytorch
pytorch-lightning 1.8.6 pypi_0 pypi
pytorch-mutex 1.0 cuda pytorch
pytz 2023.3 pypi_0 pypi
pyyaml 5.4.1 py39hb9d737c_4 conda-forge
pyzmq 25.1.0 py39h6a678d5_0
readline 8.2 h5eee18b_0
requests 2.31.0 pyhd8ed1ab_0 conda-forge
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.7.2 pyh44b312d_0 conda-forge
s2n 1.3.45 h06160fa_0 conda-forge
s3transfer 0.6.1 pyhd8ed1ab_0 conda-forge
sentry-sdk 1.25.1 pypi_0 pypi
setproctitle 1.3.2 pypi_0 pypi
setuptools 67.8.0 py39h06a4308_0
six 1.16.0 pyh6c4a22f_0 conda-forge
smmap 5.0.0 pypi_0 pypi
snappy 1.1.10 h9fff704_0 conda-forge
sqlite 3.41.2 h5eee18b_0
strenum 0.4.10 pypi_0 pypi
sympy 1.12 pyh04b8f61_3 conda-forge
tabulate 0.9.0 pypi_0 pypi
tbb 2021.9.0 hf52228f_0 conda-forge
tenacity 8.2.2 pypi_0 pypi
tensorboard 2.13.0 pypi_0 pypi
tensorboard-data-server 0.7.1 pypi_0 pypi
tensorboardx 2.6 pypi_0 pypi
termcolor 2.3.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi
torchdata 0.6.0 py39h6782a12_1 conda-forge
torchmetrics 0.11.4 pypi_0 pypi
torchtriton 2.0.0 py39 pytorch
torchvision 0.15.0 py39_cu118 pytorch
tqdm 4.65.0 pyhd8ed1ab_1 conda-forge
traitlets 5.7.1 py39h06a4308_0
typing_extensions 4.6.3 pyha770c72_0 conda-forge
tzdata 2023c h04d1e81_0
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
wandb 0.14.0 pypi_0 pypi
wcwidth 0.2.5 pyhd3eb1b0_0
werkzeug 2.3.6 pypi_0 pypi
wheel 0.38.4 py39h06a4308_0
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.4.2 h5eee18b_0
yacs 0.1.8 pypi_0 pypi
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.9.2 pypi_0 pypi
zeromq 4.3.4 h2531618_0
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
ztrbq commented
Hi magehrig, thank for your reply.
I tried both of the methods and got the same error output.
Here's my mamba list output.
# packages in environment at /home/zbq/mambaforge/envs/rvt:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
aiohttp 3.8.4 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
antlr-python-runtime 4.9.3 pyhd8ed1ab_1 conda-forge
appdirs 1.4.4 pypi_0 pypi
async-timeout 4.0.2 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
bbox-visualizer 0.1.0 pypi_0 pypi
blas 1.0 mkl conda-forge
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.19.1 hd590300_0 conda-forge
ca-certificates 2023.05.30 h06a4308_0 defaults
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
certifi 2023.5.7 pyhd8ed1ab_0 conda-forge
charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge
click 8.1.3 pypi_0 pypi
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
contourpy 1.1.0 pypi_0 pypi
cuda-cudart 11.8.89 0 nvidia
cuda-cupti 11.8.87 0 nvidia
cuda-libraries 11.8.0 0 nvidia
cuda-nvrtc 11.8.89 0 nvidia
cuda-nvtx 11.8.86 0 nvidia
cuda-runtime 11.8.0 0 nvidia
cycler 0.11.0 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
einops 0.6.0 pyhd8ed1ab_0 conda-forge
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.12.2 pyhd8ed1ab_0 conda-forge
fonttools 4.40.0 pypi_0 pypi
freetype 2.12.1 hca18f0e_1 conda-forge
frozenlist 1.3.3 pypi_0 pypi
fsspec 2023.6.0 pypi_0 pypi
gitdb 4.0.10 pypi_0 pypi
gitpython 3.1.31 pypi_0 pypi
gmp 6.2.1 h58526e2_0 conda-forge
gmpy2 2.1.2 py39h376b7d2_1 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
h5py 3.8.0 nompi_py39h89bf01e_101 conda-forge
hdf5 1.14.0 nompi_h5231ba7_103 conda-forge
hdf5plugin 4.1.2 pypi_0 pypi
hydra-core 1.3.2 pyhd8ed1ab_0 conda-forge
icu 72.1 hcb278e6_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge
intel-openmp 2023.1.0 hdb19cb5_46305 defaults
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.20.1 hf9c8cef_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.15 hfd0df8a_0 conda-forge
ld_impl_linux-64 2.38 h1181459_1 defaults
lerc 4.0.0 h27087fc_0 conda-forge
libaec 1.0.6 hcb278e6_1 conda-forge
libblas 3.9.0 1_h86c2bf4_netlib conda-forge
libbrotlicommon 1.0.9 h166bdaf_8 conda-forge
libbrotlidec 1.0.9 h166bdaf_8 conda-forge
libbrotlienc 1.0.9 h166bdaf_8 conda-forge
libcblas 3.9.0 5_h92ddd45_netlib conda-forge
libcublas 11.11.3.6 0 nvidia
libcufft 10.9.0.58 0 nvidia
libcufile 1.6.1.9 0 nvidia
libcurand 10.3.2.106 0 nvidia
libcurl 7.87.0 h6312ad2_0 conda-forge
libcusolver 11.4.1.48 0 nvidia
libcusparse 11.7.5.86 0 nvidia
libdeflate 1.17 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.4 h6a678d5_0 defaults
libgcc-ng 13.1.0 he5830b7_0 conda-forge
libgfortran-ng 13.1.0 h69a702a_0 conda-forge
libgfortran5 13.1.0 h15d22d2_0 conda-forge
libgomp 13.1.0 he5830b7_0 conda-forge
libhwloc 2.9.1 nocuda_h7313eea_6 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
liblapack 3.9.0 5_h92ddd45_netlib conda-forge
libllvm14 14.0.6 hcd5def8_3 conda-forge
libnghttp2 1.51.0 hdcd2b5c_0 conda-forge
libnpp 11.8.0.86 0 nvidia
libnsl 2.0.0 h7f98852_0 conda-forge
libnvjpeg 11.9.0.86 0 nvidia
libpng 1.6.39 h753d276_0 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libssh2 1.10.0 haa6b8db_3 conda-forge
libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge
libtiff 4.5.0 h6adf6a1_2 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.3.0 h0b41bf4_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.11.4 h0d562d8_0 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
lightning-utilities 0.8.0 pypi_0 pypi
llvmlite 0.40.0 py39h174d805_0 conda-forge
markupsafe 2.1.3 py39hd1e30aa_0 conda-forge
matplotlib 3.7.1 pypi_0 pypi
mkl 2023.1.0 h6d00ec8_46342 defaults
mpc 1.3.1 hfe3b2da_0 conda-forge
mpfr 4.2.0 hb012696_0 conda-forge
mpmath 1.3.0 pyhd8ed1ab_0 conda-forge
multidict 6.0.4 pypi_0 pypi
ncurses 6.4 h6a678d5_0 defaults
nettle 3.6 he412f7d_0 conda-forge
networkx 3.1 pyhd8ed1ab_0 conda-forge
numba 0.57.0 py39hb75a051_1 conda-forge
numpy 1.24.3 py39h6183b62_0 conda-forge
omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge
opencv-python 4.6.0.66 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 1.1.1u hd590300_0 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pandas 1.5.3 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
pillow 9.4.0 py39h2320bf1_1 conda-forge
pip 23.1.2 py39h06a4308_0 defaults
plotly 5.13.1 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pycocotools 2.0.6 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.9.15 h47a2c10_0_cpython conda-forge
python-dateutil 2.8.2 pypi_0 pypi
python_abi 3.9 3_cp39 conda-forge
pytorch 2.0.0 py3.9_cuda11.8_cudnn8.7.0_0 pytorch
pytorch-cuda 11.8 h7e8668a_5 pytorch
pytorch-lightning 1.8.6 pypi_0 pypi
pytorch-mutex 1.0 cuda pytorch
pytz 2023.3 pypi_0 pypi
pyyaml 6.0 py39hb9d737c_5 conda-forge
readline 8.2 h5eee18b_0 defaults
requests 2.31.0 pyhd8ed1ab_0 conda-forge
sentry-sdk 1.25.1 pypi_0 pypi
setproctitle 1.3.2 pypi_0 pypi
setuptools 67.8.0 py39h06a4308_0 defaults
six 1.16.0 pypi_0 pypi
smmap 5.0.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0 defaults
strenum 0.4.10 pypi_0 pypi
sympy 1.12 pypyh9d50eac_103 conda-forge
tabulate 0.9.0 pypi_0 pypi
tbb 2021.9.0 hf52228f_0 conda-forge
tenacity 8.2.2 pypi_0 pypi
tensorboardx 2.6 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 defaults
torchdata 0.6.0 py39 pytorch
torchmetrics 0.11.4 pypi_0 pypi
torchtriton 2.0.0 py39 pytorch
torchvision 0.15.0 py39_cu118 pytorch
tqdm 4.65.0 pyhd8ed1ab_1 conda-forge
typing_extensions 4.6.3 pyha770c72_0 conda-forge
tzdata 2023c h04d1e81_0 defaults
urllib3 2.0.3 pyhd8ed1ab_0 conda-forge
wandb 0.14.0 pypi_0 pypi
wheel 0.38.4 py39h06a4308_0 defaults
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.4.2 h5eee18b_0 defaults
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.9.2 pypi_0 pypi
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
robust1997 commented
I've tried your first method, the error remains. And I compare the output of conda list from no-error device and this. I install some packages that missing in the error environment, however, it doesn't fix anything. Maybe the packages' conflict caused this error? Besides, I tried to delete the entire conda environment and re-create one, it doesn't help anything.
magehrig commented
Your conda env seems to be okay. I also checked the C code, and the function here appears to return an integer smaller or equal to 0. Unfortunately, that's not helpful.
For now, I was assuming that the data you are using is not corrupted. Let's check that assumption.
- Are you using the Gen1 or Gen4 dataset for evaluation?
- post the output of
crc32 gen4.tar(assuming the gen4 dataset). It should bec5ec7c38... - Does the error above always happen with the same file (when using
hardware.num_workers.eval=0) or at least the same iteration?
If the data is not the problem, then it is either a blosc-filter bug or hdf5 bug. Then we have to find another solution.
robust1997 commented
Hi, thanks for the quick response. For me the dataset is gen4. I think the dataset is not the reason for this error, since I put this data on a hard disk and use it for different devices. The error still appears on the device which previously got bugs but doesn't appear on the device that can run this code before.
I am not sure whether this is caused by the version of the system or the hardware. To provide more info, I list them below:
The environment that has bugs: ubuntu20, cuda 11.3, 3080ti laptop gpu, 12700h cpu.
The environment that runs well: 1: ubuntu20, cuda 11.0, 3090 gpu, Intel(R) Xeon(R) Gold 6226R CPU (server)
2. ubuntu20, cuda 11.4, 2080ti gpu, 9400 cpu
Both of them have the same driver of Nvidia (530)
I will try another version of hdf5 to see if this can solve this problem.
ztrbq commented
Thanks
I use the Gen1 data downloaded from here, which is provided by the readme file.
However, when I use part of the data instead full of it, the program runs well. I tried running, for example, 36 files in validation set, the output is
Using 16bit native Automatic Mixed Precision (AMP)
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.model_summary.ModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
creating streaming val datasets: 36it [00:00, 715.60it/s]
num_full_sequences=36
num_splits=0
num_split_sequences=0
Restoring states from the checkpoint path at rvt-t.ckpt
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Loaded model weights from checkpoint at rvt-t.ckpt
/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:224: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 20 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
Validation DataLoader 0: : 0it [00:00, ?it/s]/home/zbq/mambaforge/envs/rvt/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1678402412426/work/aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Validation DataLoader 0: : 231it [01:25, 2.72it/s]creating index...
index created!
Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.54s).
Accumulating evaluation results...
DONE (t=0.11s).
Validation DataLoader 0: : 231it [01:25, 2.69it/s]
────────────────────────────────────────────────────────────────────────────────
Validate metric DataLoader 0
────────────────────────────────────────────────────────────────────────────────
val/AP 0.40103342016940047
val/AP_50 0.5584703582445278
val/AP_75 0.4436392941080782
val/AP_L 0.45298298618470323
val/AP_M 0.5116788413457906
val/AP_S 0.31680823771642735
────────────────────────────────────────────────────────────────────────────────
== Timing statistics ==
I'll try running the code on the server instead of my laptop. It seems that the problem comes with the limitation of hardware.
ztrbq commented
Hi, thanks for the quick response. For me the dataset is gen4. I think the dataset is not the reason for this error, since I put this data on a hard disk and use it for different devices. The error still appears on the device which previously got bugs but doesn't appear on the device that can run this code before. I am not sure whether this is caused by the version of the system or the hardware. To provide more info, I list them below: The environment that has bugs: ubuntu20, cuda 11.3, 3080ti laptop gpu, 12700h cpu. The environment that runs well: 1: ubuntu20, cuda 11.0, 3090 gpu, Intel(R) Xeon(R) Gold 6226R CPU (server) 2. ubuntu20, cuda 11.4, 2080ti gpu, 9400 cpu Both of them have the same driver of Nvidia (530) I will try another version of hdf5 to see if this can solve this problem.
It seems the problem occur to us both when we use the laptop. :)
magehrig commented
I see. Could be that you need more memory on your system. Indeed, this function here in the blosc library returns 0 or -1. One possibility is that you don't have enough space for decompression.
If you insist on using your laptop, you can monitor your RAM and check if you run out of memory at some point.
robust1997 commented
Hi, for me the RAM and ROM are not the limitation, my RAM is 64G and there is still around 300G free space for my SSD. Besides, I tried to only use one data file to validate, but the error remains.
Currently, I am not going to spend more time on this bug. If I get the solution in the future I will put a comment under this issue.
Thanks again for your help and for this great work:)
ztrbq commented
The error occurred again on my laptop, and I tried to use the same file as last time when the program ran well, but the error remains. I'll put the code on my server and give up my laptop.
Thanks for the help both. And thanks for the outstanding work.
magehrig commented
@robust1997 and @ztrbq, if you find out the cause, definitely let us know here.
A brute-force workaround would be to pre-process the dataset yourself but removing the blosc compression option here.
robust1997 commented
Hi, if I use my laptop preprocess this gen4 data, the error is:
name: stacked_histogram
nbins: 10
count_cutoff: 10
event_window_extraction:
method: DURATION
value: 50
fastmode: true
sequences: 0%| | 0/3 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/yuheng/anaconda3/envs/rvt/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 574, in process_sequence
write_event_data(in_h5_file=in_h5_file,
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 452, in write_event_data
write_event_representations(in_h5_file=in_h5_file,
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 496, in write_event_representations
H5Writer(ev_outfile_in_progress,
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 84, in __init__
self.h5f.create_dataset(key, dtype=self.numpy_dtype.name, shape=chunkshape, chunks=chunkshape,
File "/home/yuheng/anaconda3/envs/rvt/lib/python3.9/site-packages/h5py/_hl/group.py", line 183, in create_dataset
dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
File "/home/yuheng/anaconda3/envs/rvt/lib/python3.9/site-packages/h5py/_hl/dataset.py", line 106, in make_new_dset
dcpl = filters.fill_dcpl(
File "/home/yuheng/anaconda3/envs/rvt/lib/python3.9/site-packages/h5py/_hl/filters.py", line 281, in fill_dcpl
raise ValueError("Unknown compression filter number: %s" % compression)
ValueError: Unknown compression filter number: 32001
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 788, in <module>
for _ in pool.imap_unordered(func, iterable=seq_data_list, chunksize=chunksize):
File "/home/yuheng/anaconda3/envs/rvt/lib/python3.9/multiprocessing/pool.py", line 870, in next
raise value
ValueError: Unknown compression filter number: 32001I also try remove the blosc compression option. The command is:
self.h5f.create_dataset(key, dtype=self.numpy_dtype.name, shape=chunkshape, chunks=chunkshape,
maxshape=maxshape)The output is:
Traceback (most recent call last):
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 792, in <module>
process_sequence(dataset=dataset,
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 574, in process_sequence
write_event_data(in_h5_file=in_h5_file,
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 452, in write_event_data
write_event_representations(in_h5_file=in_h5_file,
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 505, in write_event_representations
ev_ts_us = h5_reader.time
File "/home/yuheng/Tsinghua/RVT/scripts/genx/preprocess_dataset.py", line 155, in time
self.all_times = np.asarray(self.h5f['events']['t'])
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/yuheng/anaconda3/envs/rvt/lib/python3.9/site-packages/h5py/_hl/dataset.py", line 1073, in __array__
self.read_direct(arr)
File "/home/yuheng/anaconda3/envs/rvt/lib/python3.9/site-packages/h5py/_hl/dataset.py", line 1034, in read_direct
self.id.read(mspace, fspace, dest, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5d.pyx", line 240, in h5py.h5d.DatasetID.read
File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't synchronously read data (required filter 'blosc' is not registered)magehrig commented
ah right, the original dataset is also encoded with the blosc hdf5 plugin, so you need to make sure that you use the same conda environment as the one in the readme instructions. Not sure if reading from the original dataset h5 files gives you the same errors, but you can give it a try.
robust1997 commented
Hi, this error occurs in the same environment introduced in the readme:). Do you mean trying to read data from origin .dat file?
magehrig commented
Can you post your conda list output?
The error above suggests that the hdf5 blosc plugin is not installed. Do you know if this error (required filter 'blosc' is not registered) occurred before?
robust1997 commented
The conda list output is the same as what was posted before:) This error also happens when I try to validate or train the RVT model(same error output). Maybe my expression before is a little bit misunderstood. My error is about "blosc", but the detailed error info is.
required filter 'blosc' is not registered
not exactly same as what happens on @ztrbq's error output
Here is the output of my conda list:
_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_kmp_llvm conda-forge absl-py 1.4.0 pypi_0 pypi aiohttp 3.8.4 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi antlr-python-runtime 4.9.3 pyhd8ed1ab_1 conda-forge appdirs 1.4.4 pypi_0 pypi asttokens 2.2.1 pyhd8ed1ab_0 conda-forge async-timeout 4.0.2 pypi_0 pypi attrs 23.1.0 pypi_0 pypi aws-c-auth 0.6.28 hccec9ca_5 conda-forge aws-c-cal 0.5.27 hf85dbcb_0 conda-forge aws-c-common 0.8.20 hd590300_0 conda-forge aws-c-compression 0.2.17 h4b87b72_0 conda-forge aws-c-event-stream 0.3.0 hc5de78f_6 conda-forge aws-c-http 0.7.8 h412fb1b_4 conda-forge aws-c-io 0.13.26 h0d05201_0 conda-forge aws-c-mqtt 0.8.13 ha5d9b87_2 conda-forge aws-c-s3 0.3.4 h95e21fb_5 conda-forge aws-c-sdkutils 0.1.10 h4b87b72_0 conda-forge aws-checksums 0.1.16 h4b87b72_0 conda-forge aws-crt-cpp 0.20.2 h5289e1f_9 conda-forge aws-sdk-cpp 1.10.57 h8101662_14 conda-forge awscli 1.27.153 py39hf3d152e_0 conda-forge backcall 0.2.0 pyhd3eb1b0_0 backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd3eb1b0_0 bbox-visualizer 0.1.0 pypi_0 pypi black 23.3.0 pypi_0 pypi blas 1.0 mkl blosc 1.21.4 h0f2a231_0 conda-forge blosc-hdf5-plugin 1.0.0 h91a81c6_5 conda-forge botocore 1.29.153 pyhd8ed1ab_0 conda-forge brotlipy 0.7.0 py39hb9d737c_1005 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge ca-certificates 2023.05.30 h06a4308_0 cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pypi_0 pypi certifi 2023.5.7 py39h06a4308_0 cffi 1.15.1 py39he91dace_3 conda-forge charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge click 8.1.3 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi colorama 0.4.4 pyhd3eb1b0_0 comm 0.1.3 pyhd8ed1ab_0 conda-forge contourpy 1.1.0 pypi_0 pypi cryptography 41.0.1 py39hd4f0224_0 conda-forge cuda-cudart 11.8.89 0 nvidia cuda-cupti 11.8.87 0 nvidia cuda-libraries 11.8.0 0 nvidia cuda-nvrtc 11.8.89 0 nvidia cuda-nvtx 11.8.86 0 nvidia cuda-runtime 11.8.0 0 nvidia cycler 0.11.0 pypi_0 pypi debugpy 1.6.7 py39h227be39_0 conda-forge decorator 5.1.1 pyhd3eb1b0_0 detectron2 0.6 pypi_0 pypi docker-pycreds 0.4.0 pypi_0 pypi docutils 0.16 py39hf3d152e_3 conda-forge einops 0.6.0 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge ffmpeg 4.3 hf484d3e_0 pytorch filelock 3.12.2 pyhd8ed1ab_0 conda-forge fonttools 4.40.0 pypi_0 pypi freetype 2.12.1 hca18f0e_1 conda-forge frozenlist 1.3.3 pypi_0 pypi fsspec 2023.6.0 pypi_0 pypi fvcore 0.1.5.post20221221 pypi_0 pypi gitdb 4.0.10 pypi_0 pypi gitpython 3.1.31 pypi_0 pypi gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge google-auth 2.20.0 pypi_0 pypi google-auth-oauthlib 1.0.0 pypi_0 pypi grpcio 1.54.2 pypi_0 pypi h5py 3.8.0 nompi_py39h89bf01e_101 conda-forge hdf5 1.14.0 nompi_hb72d44e_103 conda-forge hydra-core 1.3.2 pyhd8ed1ab_0 conda-forge icu 72.1 hcb278e6_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.6.0 pypi_0 pypi importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge iopath 0.1.9 pypi_0 pypi jedi 0.18.2 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jmespath 1.0.1 pyhd8ed1ab_0 conda-forge jpeg 9e h0b41bf4_3 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.4 pypi_0 pypi krb5 1.20.1 h81ceb04_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.15 hfd0df8a_0 conda-forge ld_impl_linux-64 2.38 h1181459_1 lerc 4.0.0 h27087fc_0 conda-forge libaec 1.0.6 hcb278e6_1 conda-forge libcublas 11.11.3.6 0 nvidia libcufft 10.9.0.58 0 nvidia libcufile 1.6.1.9 0 nvidia libcurand 10.3.2.106 0 nvidia libcurl 8.1.2 h409715c_0 conda-forge libcusolver 11.4.1.48 0 nvidia libcusparse 11.7.5.86 0 nvidia libdeflate 1.17 h0b41bf4_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.4.4 h6a678d5_0 libgcc-ng 13.1.0 he5830b7_0 conda-forge libgfortran-ng 13.1.0 h69a702a_0 conda-forge libgfortran5 13.1.0 h15d22d2_0 conda-forge libhwloc 2.9.1 nocuda_h7313eea_6 conda-forge libiconv 1.17 h166bdaf_0 conda-forge libllvm14 14.0.6 hcd5def8_3 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnpp 11.8.0.86 0 nvidia libnvjpeg 11.9.0.86 0 nvidia libpng 1.6.39 h753d276_0 conda-forge libsodium 1.0.18 h7b6447c_0 libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge libtiff 4.5.0 h6adf6a1_2 conda-forge libwebp-base 1.3.0 h0b41bf4_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxml2 2.11.4 h0d562d8_0 conda-forge libzlib 1.2.13 hd590300_5 conda-forge lightning-utilities 0.8.0 pypi_0 pypi llvm-openmp 16.0.5 h4dfa4b3_0 conda-forge llvmlite 0.40.0 py39h174d805_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge markdown 3.4.3 pypi_0 pypi markupsafe 2.1.3 py39hd1e30aa_0 conda-forge matplotlib 3.7.1 pypi_0 pypi mkl 2021.4.0 h8d4b97c_729 conda-forge mkl-service 2.4.0 py39h7e14d7c_0 conda-forge mkl_fft 1.3.1 py39h0c7bc48_1 conda-forge mkl_random 1.2.2 py39hde0f152_0 conda-forge mpmath 1.3.0 pyhd8ed1ab_0 conda-forge multidict 6.0.4 pypi_0 pypi mypy-extensions 1.0.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 nest-asyncio 1.5.6 py39h06a4308_0 nettle 3.6 he412f7d_0 conda-forge networkx 3.1 pyhd8ed1ab_0 conda-forge numba 0.57.0 py39hb75a051_1 conda-forge numpy 1.24.3 py39h14f4228_0 numpy-base 1.24.3 py39h31eccc5_0 oauthlib 3.2.2 pypi_0 pypi omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge opencv-python 4.6.0.66 pypi_0 pypi openh264 2.1.1 h780b84a_0 conda-forge openjpeg 2.5.0 hfec8fc6_2 conda-forge openssl 3.1.1 hd590300_1 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 1.5.3 pypi_0 pypi parso 0.8.3 pyhd3eb1b0_0 pathspec 0.11.1 pypi_0 pypi pathtools 0.1.2 pypi_0 pypi pillow 9.4.0 py39h2320bf1_1 conda-forge pip 23.1.2 py39h06a4308_0 platformdirs 3.5.3 pypi_0 pypi plotly 5.13.1 pypi_0 pypi portalocker 2.7.0 py39hf3d152e_0 conda-forge prompt-toolkit 3.0.36 py39h06a4308_0 protobuf 3.20.3 pypi_0 pypi psutil 5.9.5 pypi_0 pypi pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3eb1b0_2 pure_eval 0.2.2 pyhd3eb1b0_0 pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.3.0 pypi_0 pypi pycocotools 2.0.6 pypi_0 pypi pycparser 2.21 pyhd3eb1b0_0 pygments 2.15.1 py39h06a4308_1 pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.0.9 pypi_0 pypi pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.16 h955ad1f_3 python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytorch 2.0.0 py3.9_cuda11.8_cudnn8.7.0_0 pytorch pytorch-cuda 11.8 h7e8668a_5 pytorch pytorch-lightning 1.8.6 pypi_0 pypi pytorch-mutex 1.0 cuda pytorch pytz 2023.3 pypi_0 pypi pyyaml 5.4.1 py39hb9d737c_4 conda-forge pyzmq 25.1.0 py39h6a678d5_0 readline 8.2 h5eee18b_0 requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pypi_0 pypi rsa 4.7.2 pyh44b312d_0 conda-forge s2n 1.3.45 h06160fa_0 conda-forge s3transfer 0.6.1 pyhd8ed1ab_0 conda-forge sentry-sdk 1.25.1 pypi_0 pypi setproctitle 1.3.2 pypi_0 pypi setuptools 67.8.0 py39h06a4308_0 six 1.16.0 pyh6c4a22f_0 conda-forge smmap 5.0.0 pypi_0 pypi snappy 1.1.10 h9fff704_0 conda-forge sqlite 3.41.2 h5eee18b_0 strenum 0.4.10 pypi_0 pypi sympy 1.12 pyh04b8f61_3 conda-forge tabulate 0.9.0 pypi_0 pypi tbb 2021.9.0 hf52228f_0 conda-forge tenacity 8.2.2 pypi_0 pypi tensorboard 2.13.0 pypi_0 pypi tensorboard-data-server 0.7.1 pypi_0 pypi tensorboardx 2.6 pypi_0 pypi termcolor 2.3.0 pypi_0 pypi tk 8.6.12 h1ccaba5_0 tomli 2.0.1 pypi_0 pypi torchdata 0.6.0 py39h6782a12_1 conda-forge torchmetrics 0.11.4 pypi_0 pypi torchtriton 2.0.0 py39 pytorch torchvision 0.15.0 py39_cu118 pytorch tqdm 4.65.0 pyhd8ed1ab_1 conda-forge traitlets 5.7.1 py39h06a4308_0 typing_extensions 4.6.3 pyha770c72_0 conda-forge tzdata 2023c h04d1e81_0 urllib3 1.26.15 pyhd8ed1ab_0 conda-forge wandb 0.14.0 pypi_0 pypi wcwidth 0.2.5 pyhd3eb1b0_0 werkzeug 2.3.6 pypi_0 pypi wheel 0.38.4 py39h06a4308_0 xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.4.2 h5eee18b_0 yacs 0.1.8 pypi_0 pypi yaml 0.2.5 h7f98852_2 conda-forge yarl 1.9.2 pypi_0 pypi zeromq 4.3.4 h2531618_0 zipp 3.15.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hd590300_5 conda-forge zstd 1.5.2 h3eb15da_6 conda-forge
robust1997 commented
Hi, I think I might find the reason caused this error. When I install the ros driver of my prophesee camera, I defined the path which might influence the hdf5 function. The command is:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export HDF5_PLUGIN_PATH=$HDF5_PLUGIN_PATH:/usr/local/hdf5/lib/plugin
When I reinstall my Ubuntu, no error appears.
magehrig commented
Thanks @robust1997 for reporting your investigations. I have not thought about paths being an issue.