preprocess memory and data loading
Opened this issue · 0 comments
nadiyashore commented
Running preprocess.py would cause it to shutdown halfway through each time at the exact same point, after this output:
[2024-05-23 16:29:50,840][root][INFO] - Normalizing tas...
[2024-05-23 16:29:50,840][root][INFO] - Computing min and max...
[2024-05-23 16:29:50,840][root][INFO] - Calculation min...
giving errors related to timeout and workers.
To solve this issue, this line in nc2pt/io.py (line 20): with xr.open_mfdataset(path, engine=engine, parallel=True, chunks="auto") as ds: what changed to: with xr.open_mfdataset(path, engine=engine, parallel=True, chunks=275) as ds:
Why?