Handle output snapshots for dynamic sized arrays
benbovy opened this issue · 2 comments
benbovy commented
We need to handle dynamically sized arrays when taking multiple snapshots during a simulation.
There are different ways to solve this:
- concatenate snapshots along the dynamically sized dimension(s): see #108
- maybe resize the output (zarr) array when writing new data, so that the final shape corresponds the the max size for each dynamically sized dimension: see #111
- create a new output variable for each snapshot (e.g,
process__varname__0
,process__varname__1
, etc.).
Each of these solutions has its own advantages and limitations. We can implement each of them.
benbovy commented
Looks like option 1 could be easily derived from option 2 by chaining stack
and dropna
on the output Dataset, e.g.,
out_ds.stack(midx_dim=('clock', 'dim')).dropna('midx_dim')
Since #102, option 2 wouldn't be that bad regarding storage (depending on chunk size vs. contiguous regions of missing values), so #108 is probably not worth after all.