cannot compute annual mean with xarray 0.14.0
Closed this issue · 4 comments
Here's a quick checklist in what to include:
-
Include a detailed description of the bug or suggestion
I've updated to xarray 0.14.0, andesmlab.resample(ds, freq='ann')no longer works. -
conda listof the conda environment you are using
Here is a subset ofconda list. More details are below fromesmlab.show_versions().
cf_units 2.0.1 py37h3010b51_1002 conda-forge
cftime 1.0.4 py37hc1659b7_0 conda-forge
esmlab 2019.4.27.post30 pypi_0 pypi
numpy 1.17.2 py37h95a1406_0 conda-forge
python 3.7.3 h33d41f4_1 conda-forge
xarray 0.14.0 py_0 conda-forge
- Minimal, self-contained copy-pastable example that generates the issue if possible.
I'm calling esmlab.resample(ds, freq='ann') on a synthetic dataset constructed with the helper function xr_ds_ex.
import cftime
import numpy as np
import numpy.matlib as npm
import xarray as xr
import esmlab
def xr_ds_ex(decode_times=True, nyrs=3, var_const=True):
"""return an example xarray.Dataset object, useful for testing functions"""
# set up values for Dataset, 4 yrs of analytic monthly values
days_1yr = np.array([31.0, 28.0, 31.0, 30.0, 31.0, 30.0, 31.0, 31.0, 30.0, 31.0, 30.0, 31.0])
time_edges = np.insert(
np.cumsum(npm.repmat(days_1yr, nyrs, 1)), 0, 0)
time_bounds_vals = np.stack((time_edges[:-1], time_edges[1:]), axis=1)
time_vals = np.mean(time_bounds_vals, axis=1)
time_vals_yr = time_vals / 365.0
if var_const:
var_vals = np.ones_like(time_vals_yr)
else:
var_vals = np.sin(np.pi * time_vals_yr) * np.exp(-0.1 * time_vals_yr)
time_units = 'days since 0001-01-01'
calendar = 'noleap'
if decode_times:
time_vals = cftime.num2date(time_vals, time_units, calendar)
time_bounds_vals = cftime.num2date(time_bounds_vals, time_units, calendar)
# create Dataset, including time_bounds
time_var = xr.DataArray(time_vals, name='time', dims='time', coords={'time':time_vals},
attrs={'bounds':'time_bounds'})
if not decode_times:
time_var.attrs['units'] = time_units
time_var.attrs['calendar'] = calendar
time_bounds = xr.DataArray(time_bounds_vals, name='time_bounds', dims=('time', 'd2'),
coords={'time':time_var})
var = xr.DataArray(var_vals, name='var_ex', dims='time', coords={'time':time_var})
ds = var.to_dataset()
ds = xr.merge((ds, time_bounds))
if decode_times:
ds.time.encoding['units'] = time_units
ds.time.encoding['calendar'] = calendar
return ds
ds = xr_ds_ex(True)
print(ds)
esmlab.resample(ds, freq='ann')
The code generated the following Traceback:
Traceback (most recent call last):
File "foo.py", line 50, in <module>
esmlab.resample(ds, freq='ann')
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/esmlab/core.py", line 780, in resample
weights=weights, method=method
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/contextlib.py", line 74, in inner
return func(*args, **kwds)
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/esmlab/core.py", line 556, in compute_ann_mean
computed_dset = dset.apply(weighted_mean_arr, wgts=wgts)
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/dataset.py", line 4140, in apply
for k, v in self.data_vars.items()
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/dataset.py", line 4140, in <dictcomp>
for k, v in self.data_vars.items()
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/esmlab/core.py", line 544, in weighted_mean_arr
darr.resample({self.time_coord_name: 'A'}).mean(dim=self.time_coord_name).notnull()
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/common.py", line 1038, in resample
restore_coord_dims=restore_coord_dims,
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/resample.py", line 174, in __init__
super().__init__(*args, **kwargs)
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/groupby.py", line 336, in __init__
full_index, first_items = self._get_index_and_items(index, grouper)
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/groupby.py", line 432, in _get_index_and_items
first_items = grouper.first_items(index)
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/resample_cftime.py", line 91, in first_items
index, self.freq, self.closed, self.label, self.base
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/resample_cftime.py", line 156, in _get_time_bins
index.min(), index.max(), freq, closed=closed, base=base
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/core/resample_cftime.py", line 268, in _get_range_edges
first = first - offset
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/coding/cftime_offsets.py", line 125, in __rsub__
return -self + other
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/coding/cftime_offsets.py", line 98, in __add__
return self.__apply__(other)
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/coding/cftime_offsets.py", line 463, in __apply__
return _shift_month(other, months, self._day_option)
File "/glade/work/klindsay/miniconda3/envs/CESM2_coup_carb_cycle_JAMES/lib/python3.7/site-packages/xarray/coding/cftime_offsets.py", line 244, in _shift_month
return date.replace(year=year, month=month, day=day, dayofwk=-1)
File "cftime/_cftime.pyx", line 1239, in cftime._cftime.datetime.replace
ValueError: Replacing the dayofyr or dayofwk of a datetime is not supported.
Output of esmlab.show_versions()
#Paste the output of esmlab.show_versions() here
INSTALLED VERSIONS
commit: None
python: 3.7.3 | packaged by conda-forge | (default, Jul 1 2019, 21:52:21)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-693.21.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
esmlab: 2019.4.27.post30
xarray: 0.14.0
pandas: 0.25.2
numpy: 1.17.2
scipy: 1.3.1
xesmf: 0.2.1
cftime: 1.0.4
dask: 2.6.0
distributed: 2.6.0
setuptools: 41.4.0
pip: 19.3.1
conda: None
pytest: None
IPython: 7.8.0
sphinx: None
This looks like a different error than you were previously getting -- didn't you show me
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'CFTimeIndex'
I got the above error trying to run ds = esmlab.resample(ds, freq='ann').mean('time').compute() in a notebook and @andersy005 said it was related to #139 (I didn't save the traceback from that error, but I can recreate it if that would be helpful)
Anderson fixed this in pydata/xarray#3430. if you install cftime=1.0.3.4 that error should not appear.
@mnlevy1981 , I get the error message that you are showing if I pass decode_times=False to xr_ds_ex, instead of True. So I'm getting different error messages if time is decoded or not.
Thanks @dcherian!
Switching to cftime=1.0.3.4 solves my problem when I pass decode_times=True.
(I might have the meaning of decoded time reversed here compared to xarray. I tend to get confused about which representation of time is viewed as being encoded vs. not.)