ESDC CCI Soil Moisture failing in Preprocessing
Opened this issue · 0 comments
AliceBalfanz commented
Executing the script for CCI Soil Moisture fails with the below error in the traceback.
python inputs-preprocess/CCI/sm/cci-sm-8d-0.25deg.py
Reading
45%|██████████████████████████████████████████████▏ | 13/29 [00:00<00:00, 25.58it/s]/home/alicebalfanz/cube-gen/ESDC/inputs-preprocess/CCI/sm/cci-sm-8d-0.25deg.py:21: RuntimeWarning: Failed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:
1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.
dataset = [xr.open_zarr(file) for file in tqdm(files)]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:01<00:00, 25.25it/s]
Traceback (most recent call last):
File "/home/alicebalfanz/cube-gen/ESDC/inputs-preprocess/CCI/sm/cci-sm-8d-0.25deg.py", line 22, in <module>
dataset = xr.concat(dataset, dim="time")
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/concat.py", line 243, in concat
return _dataset_concat(
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/concat.py", line 466, in _dataset_concat
align(*datasets, join=join, copy=False, exclude=[dim], fill_value=fill_value)
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/alignment.py", line 764, in align
aligner.align()
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/alignment.py", line 557, in align
self.reindex_all()
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/alignment.py", line 535, in reindex_all
self.results = tuple(
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/alignment.py", line 536, in <genexpr>
self._reindex_one(obj, matching_indexes)
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/alignment.py", line 520, in _reindex_one
dim_pos_indexers = self._get_dim_pos_indexers(matching_indexes)
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/alignment.py", line 486, in _get_dim_pos_indexers
indexers = obj_idx.reindex_like(aligned_idx, **self.reindex_kwargs) # type: ignore[call-arg]
File "/home/conda/deepesdl/envs/cube-gen/lib/python3.9/site-packages/xarray/core/indexes.py", line 463, in reindex_like
raise ValueError(
ValueError: cannot reindex or align along dimension 'lon' because the (pandas) index has duplicate values