ZeroDivisionError in L70 of api.py
norlandrhagen opened this issue · 5 comments
norlandrhagen commented
Hi there,
I'm having an issue when trying to rechunk a 4d zarr store. In the traceback at L70 of algorithm.py (70 headroom = max_mem // chunk_mem), I'm getting the error: ZeroDivisionError: integer division or modulo by zero.
The snippet below should be a MRE.
import xarray as xr
import fsspec
import rechunker
import zarr
store_url = "gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Omon/thetao/gn/v20190429"
ds = xr.open_dataset(store_url, engine='zarr', chunks={}, decode_cf=False)
group = zarr.open_consolidated(store_url)
chunks_dict = {
'i': {'i': 128},
'j': {'j': 128},
'latitude': {'i': 128, 'j': 128},
'lev': {'lev': -1},
'lev_bnds': {'lev': -1, 'bnds': -1},
'longitude': {'i': 128, 'j': 128},
'thetao': {'i': 128, 'j': 128, 'time': 2, 'lev': -1},
'time': {'time': 2},
'time_bnds': {'time': 2, 'bnds': -1},
'vertices_latitude': {'i': 128, 'j': 128, 'vertices': -1},
'vertices_longitude': {'i': 128, 'j': 128, 'vertices': -1}
}
tmp_mapper = fsspec.get_mapper('temp_store')
tgt_mapper = fsspec.get_mapper('staging_store')
array_plan = rechunker.rechunk(group, chunks_dict, "1000MB", tgt_mapper, temp_store=tmp_mapper)A few notes:
- It seems like the
ZeroDivisionErrorhappens on thethetaovar. - In L70 of algorithm.py, in the
headroom = max_mem // chunk_memcalculation, somehow chunk_mem is being assigned to 0 in the above for loop.
chunk_mem = itemsize * prod(chunks)
if chunk_mem > max_mem:
raise ValueError(f"chunk_mem {chunk_mem} > max_mem {max_mem}")
headroom = max_mem // chunk_mem
new_chunks = list(chunks)
# only consolidate over these axes
axes = sorted(chunk_limit_per_axis.keys())[::-1]
for n_axis in axes:
c_new = min(
chunks[n_axis] * headroom, shape[n_axis], chunk_limit_per_axis[n_axis]
)
# print(f' axis {n_axis}, {chunks[n_axis]} -> {c_new}')
new_chunks[n_axis] = c_new
chunk_mem = itemsize * prod(new_chunks)
headroom = max_mem // chunk_memTraceback:
---------------------------------------------------------------------------
ZeroDivisionError Traceback (most recent call last)
Cell In[11], line 2
1 # %pdb on
----> 2 array_plan = rechunker.rechunk(group, chunks_dict, "1000MB", tgt_mapper, temp_store=tmp_mapper)
File ~/opt/anaconda3/envs/install/envs/ncviewjs/lib/python3.10/site-packages/rechunker/api.py:302, in rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options, executor)
299 if isinstance(executor, str):
300 executor = _get_executor(executor)
--> 302 copy_spec, intermediate, target = _setup_rechunk(
303 source=source,
304 target_chunks=target_chunks,
305 max_mem=max_mem,
306 target_store=target_store,
307 target_options=target_options,
308 temp_store=temp_store,
309 temp_options=temp_options,
310 )
311 plan = executor.prepare_plan(copy_spec)
312 return Rechunked(executor, plan, source, intermediate, target)
File ~/opt/anaconda3/envs/install/envs/ncviewjs/lib/python3.10/site-packages/rechunker/api.py:457, in _setup_rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options)
454 copy_specs = []
456 for array_name, array_target_chunks in target_chunks.items():
--> 457 copy_spec = _setup_array_rechunk(
...
---> 70 headroom = max_mem // chunk_mem
72 if headroom == 1:
73 break
ZeroDivisionError: integer division or modulo by zeroResults of conda list:
# Name Version Build Channel
adal 1.2.7 pyhd8ed1ab_0 conda-forge
adlfs 2022.11.2 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.3 py310h90acd4f_1 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
aiosqlite 0.18.0 pypi_0 pypi
alembic 1.9.1 pyhd8ed1ab_0 conda-forge
anyio 3.6.2 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pypi_0 pypi
appnope 0.1.3 pyhd8ed1ab_0 conda-forge
apprise 1.2.1 pypi_0 pypi
argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py310h90acd4f_3 conda-forge
asciitree 0.3.3 py_2 conda-forge
asgi-lifespan 2.0.0 pypi_0 pypi
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
asyncpg 0.27.0 py310h90acd4f_1 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
awscli 1.27.43 py310h2ec42d9_0 conda-forge
azure-core 1.26.1 pyhd8ed1ab_0 conda-forge
azure-datalake-store 0.0.51 pyh9f0ad1d_0 conda-forge
azure-identity 1.12.0 pyhd8ed1ab_0 conda-forge
azure-storage-blob 12.14.1 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bcrypt 4.0.1 pypi_0 pypi
beautifulsoup4 4.11.1 pyha770c72_0 conda-forge
bleach 5.0.1 pyhd8ed1ab_0 conda-forge
blinker 1.5 pyhd8ed1ab_0 conda-forge
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
boto3 1.26.43 pyhd8ed1ab_0 conda-forge
botocore 1.29.43 pyhd8ed1ab_0 conda-forge
brotlipy 0.7.0 py310h90acd4f_1005 conda-forge
bzip2 1.0.8 h0d85af4_4 conda-forge
c-ares 1.18.1 h0d85af4_0 conda-forge
ca-certificates 2022.12.7 h033912b_0 conda-forge
cachetools 5.2.0 pyhd8ed1ab_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cf_xarray 0.7.6 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310ha78151a_3 conda-forge
cfgv 3.3.1 pypi_0 pypi
cftime 1.6.2 py310h936d966_1 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 unix_pyhd8ed1ab_2 conda-forge
cloudpickle 2.2.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
comm 0.1.2 pyhd8ed1ab_0 conda-forge
commonmark 0.9.1 pypi_0 pypi
contourpy 1.0.6 pypi_0 pypi
coolname 2.2.0 pypi_0 pypi
croniter 1.3.8 pypi_0 pypi
cryptography 39.0.0 py310hdd0c95c_0 conda-forge
cycler 0.11.0 pypi_0 pypi
cytoolz 0.12.0 py310h90acd4f_1 conda-forge
dask 2022.12.1 pyhd8ed1ab_0 conda-forge
dask-core 2022.12.1 pyhd8ed1ab_0 conda-forge
dateparser 1.1.5 pypi_0 pypi
debugpy 1.6.5 py310h7a76584_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
dictdiffer 0.9.0 pyhd8ed1ab_0 conda-forge
distlib 0.3.6 pypi_0 pypi
distributed 2022.12.1 pyhd8ed1ab_0 conda-forge
docker 6.0.1 pypi_0 pypi
docutils 0.15.2 py310h2ec42d9_6 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
fastapi 0.88.0 pyhd8ed1ab_0 conda-forge
fasteners 0.17.3 pyhd8ed1ab_0 conda-forge
fastprogress 1.0.3 pypi_0 pypi
filelock 3.9.0 pypi_0 pypi
flit-core 3.8.0 pyhd8ed1ab_0 conda-forge
fonttools 4.38.0 pypi_0 pypi
freetype 2.12.1 h3f81eb7_1 conda-forge
frozenlist 1.3.3 py310h90acd4f_0 conda-forge
fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge
gcsfs 2022.11.0 pyhd8ed1ab_0 conda-forge
gettext 0.21.1 h8a4c099_0 conda-forge
glib 2.74.1 hbc0c0cd_1 conda-forge
glib-tools 2.74.1 hbc0c0cd_1 conda-forge
google-api-core 2.11.0 pyhd8ed1ab_0 conda-forge
google-auth 2.15.0 pyh1a96a4e_0 conda-forge
google-auth-oauthlib 0.8.0 pyhd8ed1ab_0 conda-forge
google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge
google-cloud-storage 2.7.0 pyh1a96a4e_0 conda-forge
google-crc32c 1.1.2 py310h6f7428f_4 conda-forge
google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge
googleapis-common-protos 1.57.1 pyhd8ed1ab_0 conda-forge
greenlet 2.0.1 py310h7a76584_0 conda-forge
griffe 0.25.3 pypi_0 pypi
grpcio 1.51.1 py310h0f91fd3_0 conda-forge
gst-plugins-base 1.21.3 h37e1711_1 conda-forge
gstreamer 1.21.3 h1d18e73_1 conda-forge
gunicorn 20.1.0 py310h2ec42d9_3 conda-forge
h11 0.14.0 pyhd8ed1ab_0 conda-forge
h2 4.1.0 pypi_0 pypi
heapdict 1.0.1 py_0 conda-forge
hpack 4.0.0 pypi_0 pypi
httpcore 0.16.3 pypi_0 pypi
httpx 0.23.3 pypi_0 pypi
hyperframe 6.0.1 pypi_0 pypi
icu 70.1 h96cf925_0 conda-forge
identify 2.5.12 pypi_0 pypi
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.0.0 pyha770c72_0 conda-forge
importlib_resources 5.10.2 pyhd8ed1ab_0 conda-forge
intake 0.6.6 pypi_0 pypi
intake-esm 2022.9.18 pypi_0 pypi
ipykernel 6.19.4 pyh736e0ef_0 conda-forge
ipython 8.8.0 pyhd1c38e8_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 8.0.4 pyhd8ed1ab_0 conda-forge
isodate 0.6.1 pyhd8ed1ab_0 conda-forge
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
jpeg 9e hac89ed1_2 conda-forge
jsonpatch 1.32 pypi_0 pypi
jsonpointer 2.3 pypi_0 pypi
jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge
jupyter 1.0.0 py310h2ec42d9_8 conda-forge
jupyter_client 7.4.8 pyhd8ed1ab_0 conda-forge
jupyter_console 6.4.4 pyhd8ed1ab_0 conda-forge
jupyter_core 5.1.2 py310h2ec42d9_0 conda-forge
jupyter_events 0.4.0 pyhd8ed1ab_0 conda-forge
jupyter_server 2.0.6 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.4.3 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 3.0.5 pyhd8ed1ab_0 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.20.1 h049b76e_0 conda-forge
kubernetes 25.3.0 pypi_0 pypi
lcms2 2.14 h29502cd_1 conda-forge
lerc 4.0.0 hb486fe8_0 conda-forge
libabseil 20220623.0 cxx17_h844d122_6 conda-forge
libblas 3.9.0 16_osx64_openblas conda-forge
libcblas 3.9.0 16_osx64_openblas conda-forge
libclang 13.0.1 default_he082bbe_0 conda-forge
libcrc32c 1.1.2 he49afe7_0 conda-forge
libcxx 14.0.6 hccf4f1f_0 conda-forge
libdeflate 1.14 hb7f2c08_0 conda-forge
libedit 3.1.20191231 h0678c8f_2 conda-forge
libffi 3.4.2 h0d85af4_5 conda-forge
libgfortran 5.0.0 11_3_0_h97931a8_27 conda-forge
libgfortran5 11.3.0 h082f757_27 conda-forge
libglib 2.74.1 h4c723e1_1 conda-forge
libgrpc 1.51.1 h966d1d5_0 conda-forge
libiconv 1.17 hac89ed1_0 conda-forge
libjpeg-turbo 2.1.4 hb7f2c08_0 conda-forge
liblapack 3.9.0 16_osx64_openblas conda-forge
libllvm13 13.0.1 h64f94b2_2 conda-forge
libogg 1.3.4 h35c211d_1 conda-forge
libopenblas 0.3.21 openmp_h429af6e_3 conda-forge
libopus 1.3.1 hc929b4f_1 conda-forge
libpng 1.6.39 ha978bb4_0 conda-forge
libpq 15.1 h3640bf0_2 conda-forge
libprotobuf 3.21.12 hbc0c0cd_0 conda-forge
libsodium 1.0.18 hbcb3906_1 conda-forge
libsqlite 3.40.0 ha978bb4_0 conda-forge
libtiff 4.5.0 h6268bbc_0 conda-forge
libvorbis 1.3.7 h046ec9c_0 conda-forge
libwebp-base 1.2.4 h775f41a_0 conda-forge
libxcb 1.13 h0d85af4_1004 conda-forge
libzlib 1.2.13 hfd90126_4 conda-forge
llvm-openmp 15.0.6 h61d9ccf_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.2.0 py310h8d4e1d9_0 conda-forge
lz4-c 1.9.3 he49afe7_1 conda-forge
mako 1.2.4 pyhd8ed1ab_0 conda-forge
markdown 3.4.1 pypi_0 pypi
markupsafe 2.1.1 py310h90acd4f_2 conda-forge
matplotlib 3.6.2 pypi_0 pypi
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
mistune 2.0.4 pyhd8ed1ab_0 conda-forge
msal 1.20.0 pyhd8ed1ab_0 conda-forge
msal_extensions 1.0.0 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.4 py310ha23aa8a_1 conda-forge
msrest 0.7.1 pyhd8ed1ab_0 conda-forge
multidict 6.0.4 py310h90acd4f_0 conda-forge
mypy_extensions 0.4.3 py310h2ec42d9_6 conda-forge
mysql-common 8.0.31 hc4b2c72_0 conda-forge
mysql-libs 8.0.31 h8658499_0 conda-forge
nbclassic 0.4.8 pyhd8ed1ab_0 conda-forge
nbclient 0.7.2 pyhd8ed1ab_0 conda-forge
nbconvert 7.2.7 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.2.7 pyhd8ed1ab_0 conda-forge
nbconvert-pandoc 7.2.7 pyhd8ed1ab_0 conda-forge
nbformat 5.7.1 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h96cf925_1 conda-forge
ncviewjs-backend 0.0.post28 pypi_0 pypi
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
netcdf4 1.6.2 pypi_0 pypi
nodeenv 1.7.0 pypi_0 pypi
notebook 6.5.2 pyha770c72_1 conda-forge
notebook-shim 0.2.2 pyhd8ed1ab_0 conda-forge
nspr 4.35 hea0b92c_0 conda-forge
nss 3.78 ha8197d3_0 conda-forge
numcodecs 0.11.0 py310h7a76584_1 conda-forge
numpy 1.24.1 py310h1b7c290_0 conda-forge
oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge
openjpeg 2.5.0 h13ac156_2 conda-forge
openssl 3.0.7 hfd90126_1 conda-forge
orjson 3.8.5 pypi_0 pypi
packaging 22.0 pyhd8ed1ab_0 conda-forge
pandas 1.5.2 py310hecf8f37_0 conda-forge
pandoc 2.19.2 h694c41f_1 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
paramiko 2.12.0 pypi_0 pypi
parso 0.8.3 pyhd8ed1ab_0 conda-forge
partd 1.3.0 pyhd8ed1ab_0 conda-forge
pathspec 0.10.3 pypi_0 pypi
pcre2 10.40 h1c4e4bc_0 conda-forge
pendulum 2.1.2 pypi_0 pypi
pexpect 4.8.0 pyh1a96a4e_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.4.0 py310hab5364c_0 conda-forge
pip 22.3.1 pyhd8ed1ab_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
platformdirs 2.6.2 pyhd8ed1ab_0 conda-forge
ply 3.11 py_1 conda-forge
portalocker 2.6.0 py310h2ec42d9_1 conda-forge
pre-commit 2.21.0 pypi_0 pypi
prefect 2.7.7 pypi_0 pypi
prometheus_client 0.15.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.36 pyha770c72_0 conda-forge
prompt_toolkit 3.0.36 hd8ed1ab_0 conda-forge
protobuf 4.21.12 py310h7a76584_0 conda-forge
psutil 5.9.4 py310h90acd4f_0 conda-forge
psycopg2 2.9.3 py310h31f19fa_2 conda-forge
pthread-stubs 0.4 hc929b4f_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydantic 1.10.4 py310h90acd4f_0 conda-forge
pygments 2.14.0 pyhd8ed1ab_0 conda-forge
pyjwt 2.6.0 pyhd8ed1ab_0 conda-forge
pynacl 1.5.0 pypi_0 pypi
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pypi_0 pypi
pyqt 5.15.7 py310hdd03f62_2 conda-forge
pyqt5-sip 12.11.0 py310h415000c_2 conda-forge
pyrsistent 0.19.3 py310h90acd4f_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.10.8 he7542f4_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.16.2 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.4 pyhd8ed1ab_0 conda-forge
python-slugify 7.0.0 pypi_0 pypi
python_abi 3.10 3_cp310 conda-forge
pytz 2022.7 pyhd8ed1ab_0 conda-forge
pytz-deprecation-shim 0.1.0.post0 pypi_0 pypi
pytzdata 2020.1 pypi_0 pypi
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 5.4.1 py310h90acd4f_4 conda-forge
pyzmq 24.0.1 py310hf615a82_1 conda-forge
qt-main 5.15.6 haeff654_5 conda-forge
qtconsole 5.4.0 pyhd8ed1ab_0 conda-forge
qtconsole-base 5.4.0 pyha770c72_0 conda-forge
qtpy 2.3.0 pyhd8ed1ab_0 conda-forge
re2 2022.06.01 hb486fe8_1 conda-forge
readchar 4.0.3 pypi_0 pypi
readline 8.1.2 h3899abd_0 conda-forge
rechunker 0.5.0 pyhd8ed1ab_0 conda-forge
regex 2022.10.31 pypi_0 pypi
requests 2.28.1 pyhd8ed1ab_1 conda-forge
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rfc3986 1.5.0 pypi_0 pypi
rich 13.0.0 pypi_0 pypi
rsa 4.7.2 pyh44b312d_0 conda-forge
ruamel.yaml 0.17.21 py310h90acd4f_2 conda-forge
ruamel.yaml.clib 0.2.7 py310h90acd4f_1 conda-forge
s3fs 0.4.2 py_0 conda-forge
s3transfer 0.6.0 pyhd8ed1ab_0 conda-forge
send2trash 1.8.0 pyhd8ed1ab_0 conda-forge
setuptools 65.6.3 pyhd8ed1ab_0 conda-forge
sip 6.7.5 py310h7a76584_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
skyplane 0.2.1 pypi_0 pypi
sniffio 1.3.0 pyhd8ed1ab_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge
sqlalchemy 1.4.41 py310h90acd4f_0 conda-forge
sqlalchemy2-stubs 0.0.2a31 pyhd8ed1ab_0 conda-forge
sqlite 3.40.0 h9ae0607_0 conda-forge
sqlmodel 0.0.8 pyhd8ed1ab_0 conda-forge
sshtunnel 0.4.0 pypi_0 pypi
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
starlette 0.22.0 pyhd8ed1ab_0 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
terminado 0.17.1 pyhd1c38e8_0 conda-forge
text-unidecode 1.3 pypi_0 pypi
tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h5dbffcc_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.2 py310h90acd4f_1 conda-forge
traitlets 5.8.0 pyhd8ed1ab_0 conda-forge
typer 0.7.0 pypi_0 pypi
typing-extensions 4.4.0 hd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022.7 pypi_0 pypi
tzlocal 4.2 pypi_0 pypi
universal_pathlib 0.0.21 pyh191b570_0 conda-forge
urllib3 1.26.13 pyhd8ed1ab_0 conda-forge
uvicorn 0.20.0 py310h2ec42d9_1 conda-forge
virtualenv 20.17.1 pypi_0 pypi
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 1.4.2 pyhd8ed1ab_0 conda-forge
wheel 0.38.4 pyhd8ed1ab_0 conda-forge
widgetsnbextension 4.0.5 pyhd8ed1ab_0 conda-forge
xarray 2022.12.0 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.9 h35c211d_0 conda-forge
xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge
xz 5.2.6 h775f41a_0 conda-forge
yaml 0.2.5 h0d85af4_2 conda-forge
yarl 1.8.2 py310h90acd4f_0 conda-forge
zarr 2.13.3 pyhd8ed1ab_0 conda-forge
zeromq 4.3.4 he49afe7_1 conda-forge
zict 2.2.0 pyhd8ed1ab_0 conda-forge
zipp 3.11.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hfd90126_4 conda-forge
zstd 1.5.2 hfa58983_4 conda-forge
Wondering if anyone has any thoughts on why this might be happening.
Thanks!
rabernat commented
Thanks for sharing. Can reproduce it even more simply with
target_chunks = (2, -1, 128, 128)
rechunker.rechunk(group['thetao'], target_chunks, "1000MB", tgt_mapper, temp_store=tmp_mapper)I'll investigate.
rabernat commented
So it looks like we can fix it with
target_chunks = (2, 45, 128, 128)My impression is that we don't support the syntax -1 to mean "the full size of the dimension." This could easily be fixed. Or we could raise an error if you pass a chunk shape as a negative number.
norlandrhagen commented
Thanks for the wildly quick response @rabernat!
Didn't realize the -1 means the full size on the dim wasn't supported. Either of those options sounds like an improvement to me!
rabernat commented
PR welcome! 😉
The check would go here:
rechunker/rechunker/algorithm.py
Line 68 in 46db807
Just replace the special value of -1 with the corresponding shape (i.e. swap -1 for 45 like I did above), something like
new_chunks = [s if c == -1 else c for c, s in zip(chunks, shape)]norlandrhagen commented
Sounds great! I'll open up a PR.