long pre-processing time
Opened this issue · 0 comments
Jarrett Starr and I are working on incorporating the TC MSE diagnostics into the framework code. In testing very small, simple pieces of our diagnostics, we are experiencing what appears to be a very long pre-processing time.
Jarrett ran the attached sample code that makes and saves two simple colored 2-D plots and one netCDF file through the framework. He found that it took 3 minutes and 30 seconds to get through reading in the 3 necessary data files (one 4-D (x,y,z,t) ~28 GB file and two 3-D (x,y,t) files, each ~1.5 GB) and then only took 5 more seconds to get through the python file so in total it took 3 minutes and 35 seconds.
When running the same code (but slightly altered just to account for the difference in reading in the files) locally via interactive python, it took in total 2.1 seconds.
Note that this sample code is much much shorter and simpler than what our full code will be, but it demonstrates the basic step that will then need to be repeated many times. The full code for our diagnostic will require opening three 4-D files and fourteen 3-D files ~thousands of times, as we loop over each 6-hourly position of each tracked tropical cyclone to extract the necessary data in a 10 x 10 degree box around the tropical cyclone and compute the diagnostic (eventually all the storm snapshots are composited together in various ways). So, we are concerned that the pre-processor overhead will cause the POD to take a prohibitively long time to run (and even, a prohibitively long time for us to test it within the framework!)
The attached zip file contains the config.jsonc file, settings.jsonc file for our test diagnostic, .py file for our test diagnostic, and .log file for a run of our test diagnostic. The three data files that are used in this test are too big to attach but I can provide globus access if needed. They are actually AM4 files that @jkrasting provided us so perhaps he still has them:
GFDL.CM4.AMIP.1979010100-1983123123.rlds.6hr.nc
GFDL.CM4.AMIP.1979010100-1983123123.rlus.6hr.nc
GFDL.CM4.AMIP.1979010100-1983123123.ta.6hr.nc
@jkrasting and @wrongkindofdoctor suspected that this was related to the preprocessor overhead of rewriting the input files to the work directory.
Finally, I note that this is an issue for our TC MSE POD, but it will also be an issue for our new tropical disturbance POD, which will operate on different weather systems in a similar fashion.