Performance issue with nudging with FMS2

[yichengt900]

Our regional seasonal prediction experiments have reported significantly slower run times when running MOM6 with nudging using FMS2:

With FMS1:

                                       hits          tmin          tmax             tavg                tstd           tfrac    grain pemin pemax
(Ocean sponges)                         48      1.046542      3.557714      1.843401      0.315346  0.008    31             0     1645

With FMS2:

                                       hits          tmin          tmax                tavg                tstd           tfrac    grain pemin pemax
(Ocean sponges)                         48    197.205940    199.744052    198.060822      0.332912  0.402    31     0  1645

After further investigation, we found that the performance difference arises from the MOM_interp_infra.F90 file, which differs between FMS1 and FMS2:

• FMS2 uses get_var_axes_info from MOM_io.f90, which opens and closes a NetCDF file at every time step, making both operations slow.
• FMS1 uses get_extern_field_axes from time_interp_external.f90. which does not open, close, or read NetCDF files at every time step; this is handled during the `init_external_field' step.

To address this on the regional side, we have added a simple logic as a temporary fix to ensure get_var_axes_info runs only during the first time step. However, there may be a better way to handle this issue."

https://github.com/NOAA-CEFI-Regional-Ocean-Modeling/MOM6/blob/4a3c5e0dbbc8664fdddd9893c43b7c0985b71ef6/src/framework/MOM_horizontal_regridding.F90#L739-L742