ncar-xdev/xpersist

`CacheStore.delete()` doesn't work properly when deleting keys corresponding to directories

Closed this issue · 0 comments

When using .delete() method to delete a key corresponding to a directory (e.g. a Zarr store), xpersist runs into different kinds of errors (PermissionError, IsADirectoryError). This issue stems from the fsspec mapper delitems() method used in : https://github.com/NCAR/xpersist/blob/d1e8a2154dec844021a7fa1a3428bed8944f75bc/xpersist/cache.py#L117. We should switch to fsspec's delete method instead. This will allow us to recursively delete keys corresponding to directories....

In [1]: import xarray as xr, xpersist as xp

In [2]: ds = xr.tutorial.open_dataset('rasm')

In [3]: store = xp.CacheStore()

In [4]: store.keys()
Out[4]: []

In [5]: _ = store.put('foo.zarr', ds, serializer='xarray.zarr')

In [6]: store.keys()
Out[6]: ['foo.zarr']
In [7]: store.delete('foo.zarr')
DRY RUN: would delete items with keys: ['foo.zarr', 'xpersist_metadata_store/foo.zarr.artifact.json']

In [8]: store.delete('foo.zarr', dry_run=False)
---------------------------------------------------------------------------
IsADirectoryError                         Traceback (most recent call last)
<ipython-input-8-5acbab83d9b0> in <module>
----> 1 store.delete('foo.zarr', dry_run=False)

/glade/work/abanihi/opt/miniconda/envs/playground/lib/python3.8/site-packages/xpersist/cache.py in delete(self, key, dry_run)
    115         keys = [key, self._artifact_meta_relative_path(key)]
    116         if not dry_run:
--> 117             self.mapper.delitems(keys)
    118         else:
    119             print(f'DRY RUN: would delete items with keys: {repr(keys)}')

/glade/work/abanihi/opt/miniconda/envs/playground/lib/python3.8/site-packages/fsspec/mapping.py in delitems(self, keys)
    115     def delitems(self, keys):
    116         """Remove multiple keys from the store"""
--> 117         self.fs.rm([self._key_to_str(k) for k in keys])
    118 
    119     def _key_to_str(self, key):

/glade/work/abanihi/opt/miniconda/envs/playground/lib/python3.8/site-packages/fsspec/implementations/local.py in rm(self, path, recursive, maxdepth)
    147                 shutil.rmtree(p)
    148             else:
--> 149                 os.remove(p)
    150 
    151     def _open(self, path, mode="rb", block_size=None, **kwargs):

IsADirectoryError: [Errno 21] Is a directory: '/glade/scratch/abanihi/foo.zarr'