Distribute spatial chunks for forward passes across nodes?
bnb32 opened this issue · 1 comments
Why this feature is necessary:
Memory limits. In theory the spatial grid of the forward pass output could be so large that not even 1 timestep can be held in memory on a single node. (And reducing the forward pass size to a single time step creates other issues with the overlapping motivation).
A possible solution is:
I think the easiest would be to include the write_output calls in the loops over the forward pass chunks and just write each chunk output to its own file, instead of combing chunks after the fp loops and then writing the combined array.
Urgency / Timeframe
This probably wont be needed for quite a while. I am currently able to reduce fwp_chunk_size[2] to run the pipeline on the hourly 4km new wtk data but if we were doing much more than a 2x spatial enhancement on the full grid this would be a problem.
This is resolved with the fwp_spatial_dist branch