xarray-contrib/xarray-simlab

Handle output snapshots for dynamic sized arrays

benbovy opened this issue · 2 comments

We need to handle dynamically sized arrays when taking multiple snapshots during a simulation.

There are different ways to solve this:

  1. concatenate snapshots along the dynamically sized dimension(s): see #108
  2. maybe resize the output (zarr) array when writing new data, so that the final shape corresponds the the max size for each dynamically sized dimension: see #111
  3. create a new output variable for each snapshot (e.g, process__varname__0, process__varname__1, etc.).

Each of these solutions has its own advantages and limitations. We can implement each of them.

Looks like option 1 could be easily derived from option 2 by chaining stack and dropna on the output Dataset, e.g.,

out_ds.stack(midx_dim=('clock', 'dim')).dropna('midx_dim')

Since #102, option 2 wouldn't be that bad regarding storage (depending on chunk size vs. contiguous regions of missing values), so #108 is probably not worth after all.

Closed in #111