Unable to load `np.load(...)` your provided `npy` dataset. allow_pickle not working

[jpainam]

Hi, when I try to load your npy data using np.load, it throws this error.
Even after passing allow_pickle=True to np.load(...), I still face the same problem

How did you load your npy files?
Thanks

Traceback (most recent call last):
  File "main_diffusion.py", line 28, in <module>
    app.run(main)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "main_diffusion.py", line 21, in main
    trainer.train(FLAGS.config)
  File "~/MeshDiffusion/lib/diffusion/trainer.py", line 100, in train
    batch = next(data_iter)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/gpfs/u/scratch/QCML/shared/mesh/MeshDiffusion/lib/dataset/shapenet_dmtet_dataset.py", line 39, in __getitem__
    datum = torch.tensor(np.load(self.fpath_list[idx]))
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/numpy/lib/npyio.py", line 438, in load
    raise ValueError("Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False

when passing allow_pickle=True, This is the error

dataset  with sdf normalized: True
data loader set
I0222 11:28:01.073143 140650419656576 trainer.py:92] Starting training loop at step 0.
Traceback (most recent call last):
  File "main_diffusion.py", line 28, in <module>
    app.run(main)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "main_diffusion.py", line 21, in main
    trainer.train(FLAGS.config)
  File "/gpfs/u/scratch/QCML/shared/mesh/MeshDiffusion/lib/diffusion/trainer.py", line 100, in train
    batch = next(data_iter)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
_pickle.UnpicklingError: Caught UnpicklingError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/numpy/lib/npyio.py", line 441, in load
    return pickle.load(fid, **pickle_kwargs)
_pickle.UnpicklingError: invalid load key, 'v'.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/gpfs/u/scratch/QCML/shared/mesh/MeshDiffusion/lib/dataset/shapenet_dmtet_dataset.py", line 39, in __getitem__
    datum = torch.tensor(np.load(self.fpath_list[idx], allow_pickle=True))
  File "~/miniconda3x86/envs/mesh/lib/python3.8/site-packages/numpy/lib/npyio.py", line 443, in load
    raise pickle.UnpicklingError(
_pickle.UnpicklingError: Failed to interpret file '~/MeshDiffusion/dmtet_npy_folder/res64/chair/4618.npy' as a pickle

[jpainam]

In case you faced the same issue, this is how I resolved the issue:

Install lfs using brew install git-lfs, then git lfs install on MacOS or download the binary for your OS https://git-lfs.com/
Then run git clone git@hf.co:datasets//lzzcd001/MeshDiffusion_DMTet_Dataset to download the dataset.

Without git lfs install, you get corrupted dataset.