`CacheStore.delete()` doesn't work properly when deleting keys corresponding to directories
Closed this issue · 0 comments
andersy005 commented
When using .delete()
method to delete a key corresponding to a directory (e.g. a Zarr store), xpersist runs into different kinds of errors (PermissionError,
IsADirectoryError
). This issue stems from the fsspec mapper delitems()
method used in : https://github.com/NCAR/xpersist/blob/d1e8a2154dec844021a7fa1a3428bed8944f75bc/xpersist/cache.py#L117. We should switch to fsspec's delete
method instead. This will allow us to recursively delete keys corresponding to directories....
In [1]: import xarray as xr, xpersist as xp
In [2]: ds = xr.tutorial.open_dataset('rasm')
In [3]: store = xp.CacheStore()
In [4]: store.keys()
Out[4]: []
In [5]: _ = store.put('foo.zarr', ds, serializer='xarray.zarr')
In [6]: store.keys()
Out[6]: ['foo.zarr']
In [7]: store.delete('foo.zarr')
DRY RUN: would delete items with keys: ['foo.zarr', 'xpersist_metadata_store/foo.zarr.artifact.json']
In [8]: store.delete('foo.zarr', dry_run=False)
---------------------------------------------------------------------------
IsADirectoryError Traceback (most recent call last)
<ipython-input-8-5acbab83d9b0> in <module>
----> 1 store.delete('foo.zarr', dry_run=False)
/glade/work/abanihi/opt/miniconda/envs/playground/lib/python3.8/site-packages/xpersist/cache.py in delete(self, key, dry_run)
115 keys = [key, self._artifact_meta_relative_path(key)]
116 if not dry_run:
--> 117 self.mapper.delitems(keys)
118 else:
119 print(f'DRY RUN: would delete items with keys: {repr(keys)}')
/glade/work/abanihi/opt/miniconda/envs/playground/lib/python3.8/site-packages/fsspec/mapping.py in delitems(self, keys)
115 def delitems(self, keys):
116 """Remove multiple keys from the store"""
--> 117 self.fs.rm([self._key_to_str(k) for k in keys])
118
119 def _key_to_str(self, key):
/glade/work/abanihi/opt/miniconda/envs/playground/lib/python3.8/site-packages/fsspec/implementations/local.py in rm(self, path, recursive, maxdepth)
147 shutil.rmtree(p)
148 else:
--> 149 os.remove(p)
150
151 def _open(self, path, mode="rb", block_size=None, **kwargs):
IsADirectoryError: [Errno 21] Is a directory: '/glade/scratch/abanihi/foo.zarr'